Business entities see arbitration as a forum where they can settle their disputes. This form of alternative dispute resolution provides a win-win situation for both parties involved in the debate. Arbitration is the only institution that has full authority to settle their disputes once parties entered into a consent to choose an arbitration committee over a classic judicial forum. Even though arbitral awards have a final and binding character, they may be challenged using two legal methods: refusal or annulment. Besides providing specific grounds of refusal, the New York Convention 1958 ruled that the annulment of a foreign arbitral award could be done by a “competent authority of the country in which, or under the law of which, the award was made”. Although Indonesia has ratified the Convention and has specific national regulations on arbitration, judges of the courts of the first and second instance in fact do not have sufficient understanding of the refusal and annulment grounds of foreign arbitral awards. The well-known case of Karaha Bodas Company v. Pertamina shows that judges of the District Court of Central Jakarta failed to exercise their jurisdiction to annul the Geneva arbitral award. The case is a very typical example of not thinking outside the box and disregarding international treaties that sadly seems to be a commonly followed ‘habit’ in many cases all over the world. This paper aims to criticize several mistakes in the judicial reasoning that lead to such outcomes in judicial practice. Moreover, this paper will explain ways to strengthen judges’ ability to understand the international treaties ratified by their government, as their usual practice in the civil law system is to constantly rely on the hierarchy of national legislation.