Asset variables needs [SerializeField] to be saved when reopening Unity.

Is this the intended behaviour? I’ve been creating my own custom Editor Windows, with the help of which I edit the ScriptableObject asset.

I do know about the EditorUtility.SetDirty(asset) and I use it every single time the custom editor makes changes, but when I didn’t have [SerializeField] attribute in front of the variables (and [System.Serializable] on the class), it just refused to save.

Yes this is expected. You may find the page in the docs on Script Serialization helpful.

Moreover, it is recommended that your custom editor display these fields using controls with the SerializedProperty class (via the Editor.serializedObject property) rather than directly modifying the field on the inspected target, as SerializedProperty will automatically take care of dirtying, undo/redo, multi-editing, and so on. You can search around this site for more info if you run into issues with it, but the basic pattern is something like this:

public override void OnInspectorGUI()
{
    this.serializedObject.Update();
    // your controls here, like e.g., EditorGUILayout.PropertyField(this.serializedObject.FindProperty("someField"));
    this.serializedObject.ApplyModifiedProperties();
}