The error says that the function has a parameter type of *Vector3[] (pointer to vector3 array). You are passing in a Vector3[] (vector3 array - not a pointer to one). In C#, the signature reads as params Vector3[], which means that you can pass several Vector3s separated by comma, or send a vector3 array. I think this is a design bug in Unity.

It seems that the function expects a pointer to an array of Vector3. I am not well versed in JS to know how to obtain a pointer, or if it is even possible to obtain the pointer of an array. Until someone else post a definite answer how to solve this in JS I can present you a workaround with C# code, proxying the call without the params version. Your JS code can then call this instead of Handles.

Put this in Standard Assets folder to ensure it compiles before your script.

using UnityEngine;
using UnityEditor;

/// <summary>
/// Handles eXtension.
/// Provides support to DrawAAPolyLine functions for JS.
/// </summary>
public static class HandlesX
{
    public static void DrawAAPolyLine(Vector3[] points)
    {
        Handles.DrawAAPolyLine(points);
    }
    public static void DrawAAPolyLine(float width, Vector3[] points)
    {
        Handles.DrawAAPolyLine(width, points);
    }
    public static void DrawAAPolyLine(Texture2D lineTex, Vector3[] points)
    {
        Handles.DrawAAPolyLine(lineTex, points);
    }
    public static void DrawAAPolyLine(Texture2D lineTex, float width, Vector3[] points)
    {
        Handles.DrawAAPolyLine(lineTex, width, points);
    }
}

Then you can use HandlesX.DrawAAPolyLine in a similar fashion, from your JS code.

The error is gone after I passed in the elements of the vector3 array instead of the whole array:

Handles.DrawAAPolyLine(vecArray[0], vecArray[1], vecArray[2]);

I get the error if I pass in vecArray:

Handles.DrawAAPolyLine(vecArray);

this is the function declaration:

static function DrawAAPolyLine (params points : Vector3[]) : void