Hi ,
To answer your question, the users should have access to Gateway system. When the user tries to access an OData service, he needs to enter the credentials for the gateway system, not the backend system. Now it is responsibility of the gateway to call the back end and fetch data. Gateway does that by SYSTEM alias configuration. The RFC behind the system alias should be configured to be able to call the back-end.
So, the users should be created in Gateway System.
Regards,
Atanu