8000 Load entities from local file by sdransfeld · Pull Request #4 · eblis/python-odata · GitHub
[go: up one dir, main page]
Skip to content

Load entities from local file#4

Closed
sdransfeld wants to merge 0 commit intoeblis:masterfrom
sdransfeld:master
Closed

Load entities from local file#4
sdransfeld wants to merge 0 commit intoeblis:masterfrom
sdransfeld:master

Conversation

@sdransfeld
Copy link
@sdransfeld sdransfeld commented May 16, 2024

Hi,

Some services have huge metadata. For example Dynamics 365 has over 30MB. It would be nice to be able to load the metadata from a local file.

I have tried "reflect_output_package", but it gives me an error:
File "C:\Users\104615sedr\devel\extern\python-odata\odata\service.py", line 156, in init
self._write_reflected_types(metadata_url=self.metadata_url, package=reflect_output_package)
File "C:\Users\104615sedr\devel\extern\python-odata\odata\service.py", line 166, in _write_reflected_types
outputter.write_reflected_types()
File "C:\Users\104615sedr\devel\extern\python-odata\odata\reflector.py", line 120, in write_reflected_types
template.render_context(context)
File "C:\Users\104615sedr\devel\services\build\python-3.11.9.amd64\Lib\site-packages\mako\template.py", line 455, in render_context
runtime.render_context(self, self.callable, context, *args, **kwargs)
File "C:\Users\104615sedr\devel\services\build\python-3.11.9.amd64\Lib\site-packages\mako\runtime.py", line 916, in _render_context
_exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
File "C:\Users\104615sedr\devel\services\build\python-3.11.9.amd64\Lib\site-packages\mako\runtime.py", line 943, in exec_template
callable(context, *args, **kwargs)
File "main_mako", line 41, in render_body
File "C:\Users\104615sedr\devel\services\build\python-3.11.9.amd64\Lib\site-packages\mako\runtime.py", line 793, in include_file
callable(ctx, **kwargs)
File "entity_mako", line 28, in render_body
File "C:\Users\104615sedr\devel\services\build\python-3.11.9.amd64\Lib\enum.py", line 782, in getattr
raise AttributeError(name)
AttributeError: odata_collection

@eblis
Copy link
Owner
eblis commented Oct 9, 2024

Can you show me the code how you used reflect_output_packages ?

@eblis
Copy link
Owner
eblis commented Oct 9, 2024

I am loading an 15mb persisted file without issues.

@eblis
Copy link
Owner
eblis commented Oct 9, 2024 •
edited
Loading

Here's an example how you can use it for NorthWind service:

import requests
from datetime import datetime

# you can only import this on the second run, the first run will create the package
# import generated.northwind
from odata import ODataService
session = requests.Session()

url = 'http://services.odata.org/V4/Northwind/Northwind.svc/'
service = ODataService(
    url="http://services.odata.org/V4/Northwind/Northwind.svc/",
    session=session,
    base=generated.northwind.ReflectionBase,
    reflect_entities=True,
    reflect_output_package="generated.northwind")
OrderDetails = generated.northwind.Order_Details

@sdransfeld
Copy link
Author
sdransfeld commented Oct 9, 2024 
def main():
    import requests
    from datetime import datetime

    # you can only import this on the second run, the first run will create the package
    # import generated
    from odata import ODataService
    session = requests.Session()

    url = 'https://mg-prod.operations.dynamics.com/data/'
    service = ODataService(
        url=url,
        session=session,
        # base=generated.dynamics.ReflectionBase,
        reflect_entities=True,
        reflect_output_package="generated.dynamics",
        auth=get_auth())
    # MGRProjects = generated.dynamics.MGRProjects

Traceback (most recent call last):
  File "C:\Users\104615sedr\devel\py-erp\reflect.py", line 41, in <module>
    main()
  File "C:\Users\104615sedr\devel\py-erp\reflect.py", line 30, in main
    service = ODataService(
              ^^^^^^^^^^^^^
  File "C:\Users\104615sedr\devel\services\build\python-3.11.9.amd64\Lib\site-packages\odata\service.py", line 156, in __init__
    self._write_reflected_types(metadata_url=self.metadata_url, package=reflect_output_package)
  File "C:\Users\104615sedr\devel\services\build\python-3.11.9.amd64\Lib\site-packages\odata\service.py", line 166, in _write_reflected_types
    outputter.write_reflected_types()
  File "C:\Users\104615sedr\devel\services\build\python-3.11.9.amd64\Lib\site-packages\odata\reflector.py", line 120, in write_reflected_types
    template.render_context(context)
  File "C:\Users\104615sedr\devel\services\build\python-3.11.9.amd64\Lib\site-packages\mako\template.py", line 455, in render_context
    runtime._render_context(self, self.callable_, context, *args, **kwargs)
  File "C:\Users\104615sedr\devel\services\build\python-3.11.9.amd64\Lib\site-packages\mako\runtime.py", line 916, in _render_context
    _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
  File "C:\Users\104615sedr\devel\services\build\python-3.11.9.amd64\Lib\site-packages\mako\runtime.py", line 943, in _exec_template
    callable_(context, *args, **kwargs)
  File "main_mako", line 41, in render_body
  File "C:\Users\104615sedr\devel\services\build\python-3.11.9.amd64\Lib\site-packages\mako\runtime.py", line 793, in _include_file
    callable_(ctx, **kwargs)
  File "entity_mako", line 28, in render_body
  File "C:\Users\104615sedr\devel\services\build\python-3.11.9.amd64\Lib\enum.py", line 782, in __getattr__
    raise AttributeError(name)
AttributeError: __odata_collection__

@sdransfeld
Copy link
Author
sdransfeld commented Oct 9, 2024

python-odata=0.5.4
Mako=1.3.3
python=3.11.9

@eblis
Copy link
Owner
eblis commented Oct 9, 2024

Could you share the metadata response or is that private ?
Looks like you have enums in your metadata, there's another pull request open for that, i'm currently looking into it. Would help if I had an example metadata file with enums.

@sdransfeld
Copy link
Author
sdransfeld commented Oct 9, 2024

The data is mostly standard MS Dynamics data (https://www.microsoft.com/en-us/dynamics-365)
metadata.zip

@eblis
Copy link
Owner
eblis commented Oct 9, 2024

Can you try with python-odata version 0.6.0 ? I added support for enums. I didn't test with your metadata yet, I tested with the ODATA example service, TripPin.

@jonnerd154
Copy link
jonnerd154 commented Oct 9, 2024 •
edited
Loading

I was having the same problem...freshly minted v0.6.0 fixed it for me. 🎉 I was chasing this problem all night - very happy to see this discussion (and resolution!) this morning.

THANK YOU! This is a great library - much appreciated.

(I'm working with enums galore in SAP Business One's Service Layer)

@jonnerd154
Copy link
jonnerd154 commented Oct 9, 2024 •
edited
Loading

I've reworked my includes/etc to use the reflected-to-file classes (thank you for adding enum support @eblis!). Now I'm having a different issue and not sure how best to solve it - or maybe I'm using the library incorrectly.

Using the file-reflected Classes, the entity classes don't have the __odata_service__ attribute, so Query can't retrieve the odata service base URL to form it's complete request url.

File "****************\.venv\Lib\site-packages\odata\entity.py", line 101, in __odata_url__     
    return urljoin(cls.__odata_service__.url, cls.__odata_collection__)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'url'

I patched it changing Service to pass the odata service's base url to the reflector, and down to main.mako to add an anonymous object to the ReflectionBase class called __odata_service__ with a url attribute:

class ReflectionBase(EntityBase):
    __odata_service__ = type('',(object,),{"url": "${service_url}"})()
    pass

...This solves the problem, but feels a little hacky to have the URL static in the generated class file. Thoughts?

@eblis
Copy link
Owner
eblis commented Oct 9, 2024 •
edited
Loading

Make sure you pass base from the reflected package, check the example above with NorthWind.

I made some changes related to this so it will automatically try to import and use that package for you, it's coming in 0.6.1, but only in the next days.

@jonnerd154
Copy link
jonnerd154 commented Oct 9, 2024

Make sure you pass base from the reflected package, check the example above with NorthWind.

That did it, thank you! Looking forward to your updates.

Also, I have a couple of changes that allow compatibility with services that expose entity attributes that are read-only. For example, SAP B1 Service Layer will return an error if Orders.RelatedTyp is transmitted as null when trying to insert a new record. The library initializes all the instance variables to be None, which causes issues, so those attributes need to be un-set before the request is sent. I need to do a little more testing, but I'll open a PR later this week.

Thanks again.

@eblis
Copy link
Owner
eblis commented Oct 9, 2024

Can you give me the URL for SAP service, is it public?
If not, could you share the metadata file so I can have a look at those properties as well? Are they marked in any special way?

@jonnerd154
Copy link
jonnerd154 commented Oct 9, 2024 •
edited
Loading

Happy to share the metadata file. The server/service is private, but here's nothing proprietary in the metadata file. I'll email it to you.

Edit: the properties that are causing issues are contextually read-only: sometimes it's OK to write to them, but not always, and never during an insert.

I also just opened a PR #5 with two workarounds that enable my application - you clearly know more about the odata spec, all advice/edits welcome.

@sdransfeld sdransfeld closed this Nov 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@sdransfeld @eblis @jonnerd154
0