Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: integration with dd-trace-api #12057

Open
wants to merge 36 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
eb66781
stub integration with dd-trace-api
emmettbutler Jan 22, 2025
198aaa9
a basic but valid failing test
emmettbutler Jan 23, 2025
d60b73c
fleshing out the test setup
emmettbutler Jan 24, 2025
4e1004d
working on the dummy tracer...
emmettbutler Jan 24, 2025
dae3efd
store tracer globally
emmettbutler Jan 24, 2025
3e40d48
pull from github
emmettbutler Jan 24, 2025
181b1c1
add dd_trace_api integration to CI
emmettbutler Jan 24, 2025
ffc79e5
linting
emmettbutler Jan 24, 2025
89e04a1
enable snapshots
emmettbutler Jan 24, 2025
9bc3423
doesnt work on 3.7
emmettbutler Jan 24, 2025
60ca763
stub verification
emmettbutler Jan 24, 2025
4ffec90
stub verification
emmettbutler Jan 24, 2025
b1db4dc
only hook once
emmettbutler Jan 24, 2025
d1a7353
more generic code
emmettbutler Jan 24, 2025
1b3cec2
methods on real spans to avoid bookkeeping
emmettbutler Jan 27, 2025
d3a6196
current_span APIs
emmettbutler Jan 27, 2025
026c06d
use singleton tracer
emmettbutler Jan 27, 2025
e1f1a68
convert stubs to real spans when passed to API functions
emmettbutler Jan 27, 2025
36cdb43
slight refactors
emmettbutler Jan 27, 2025
9bba449
remove some duplication from handlers list
emmettbutler Jan 27, 2025
2448b3e
remove some duplication from handlers list
emmettbutler Jan 27, 2025
35d8a49
reno
emmettbutler Jan 27, 2025
66bdb34
Merge branch 'main' into emmett.butler/dd-trace-api-integration
emmettbutler Jan 27, 2025
b787857
simplify
emmettbutler Jan 28, 2025
86d69dd
simplify
emmettbutler Jan 28, 2025
bae4cac
unused import
emmettbutler Jan 28, 2025
ee4492c
some more tests
emmettbutler Jan 28, 2025
50ce3bd
Merge branch 'main' into emmett.butler/dd-trace-api-integration
emmettbutler Jan 29, 2025
abca08f
Merge branch 'main' into emmett.butler/dd-trace-api-integration
emmettbutler Jan 31, 2025
17552de
add patching for tracer.wrap, which requires API accessing the return…
emmettbutler Feb 3, 2025
d147067
remove tests for removed functionality
emmettbutler Feb 3, 2025
db87343
remove tests
emmettbutler Feb 4, 2025
e043f22
code organization
emmettbutler Feb 4, 2025
5540bd1
no shared state
emmettbutler Feb 5, 2025
acac914
update requirements
emmettbutler Feb 5, 2025
0e679f8
one hook name
emmettbutler Feb 5, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions .riot/requirements/10e65d1.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
#
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
# pip-compile --no-annotate .riot/requirements/10e65d1.in
#
attrs==24.3.0
certifi==2024.12.14
charset-normalizer==3.4.1
coverage[toml]==7.6.10
dd-trace-api @ git+https://github.com/DataDog/dd-trace-api-py
exceptiongroup==1.2.2
hypothesis==6.45.0
idna==3.10
iniconfig==2.0.0
mock==5.1.0
opentracing==2.4.0
packaging==24.2
pluggy==1.5.0
pytest==8.3.4
pytest-cov==6.0.0
pytest-mock==3.14.0
pyyaml==6.0.2
requests==2.32.3
sortedcontainers==2.4.0
tomli==2.2.1
urllib3==2.3.0
24 changes: 24 additions & 0 deletions .riot/requirements/1261872.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
#
# This file is autogenerated by pip-compile with Python 3.12
# by the following command:
#
# pip-compile --no-annotate .riot/requirements/1261872.in
#
attrs==24.3.0
certifi==2024.12.14
charset-normalizer==3.4.1
coverage[toml]==7.6.10
dd-trace-api @ git+https://github.com/DataDog/dd-trace-api-py
hypothesis==6.45.0
idna==3.10
iniconfig==2.0.0
mock==5.1.0
opentracing==2.4.0
packaging==24.2
pluggy==1.5.0
pytest==8.3.4
pytest-cov==6.0.0
pytest-mock==3.14.0
requests==2.32.3
sortedcontainers==2.4.0
urllib3==2.3.0
27 changes: 27 additions & 0 deletions .riot/requirements/14d1688.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
#
# This file is autogenerated by pip-compile with Python 3.8
# by the following command:
#
# pip-compile --no-annotate .riot/requirements/14d1688.in
#
attrs==24.3.0
certifi==2024.12.14
charset-normalizer==3.4.1
coverage[toml]==7.6.1
dd-trace-api @ git+https://github.com/DataDog/dd-trace-api-py
exceptiongroup==1.2.2
hypothesis==6.45.0
idna==3.10
iniconfig==2.0.0
mock==5.1.0
opentracing==2.4.0
packaging==24.2
pluggy==1.5.0
pytest==8.3.4
pytest-cov==5.0.0
pytest-mock==3.14.0
pyyaml==6.0.2
requests==2.32.3
sortedcontainers==2.4.0
tomli==2.2.1
urllib3==2.2.3
27 changes: 27 additions & 0 deletions .riot/requirements/668f2f5.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
#
# This file is autogenerated by pip-compile with Python 3.9
# by the following command:
#
# pip-compile --no-annotate .riot/requirements/668f2f5.in
#
attrs==24.3.0
certifi==2024.12.14
charset-normalizer==3.4.1
coverage[toml]==7.6.10
dd-trace-api @ git+https://github.com/DataDog/dd-trace-api-py
exceptiongroup==1.2.2
hypothesis==6.45.0
idna==3.10
iniconfig==2.0.0
mock==5.1.0
opentracing==2.4.0
packaging==24.2
pluggy==1.5.0
pytest==8.3.4
pytest-cov==6.0.0
pytest-mock==3.14.0
pyyaml==6.0.2
requests==2.32.3
sortedcontainers==2.4.0
tomli==2.2.1
urllib3==2.3.0
25 changes: 25 additions & 0 deletions .riot/requirements/d5f777e.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
#
# This file is autogenerated by pip-compile with Python 3.13
# by the following command:
#
# pip-compile --no-annotate .riot/requirements/d5f777e.in
#
attrs==24.3.0
certifi==2024.12.14
charset-normalizer==3.4.1
coverage[toml]==7.6.10
dd-trace-api @ git+https://github.com/DataDog/dd-trace-api-py
hypothesis==6.45.0
idna==3.10
iniconfig==2.0.0
mock==5.1.0
opentracing==2.4.0
packaging==24.2
pluggy==1.5.0
pytest==8.3.4
pytest-cov==6.0.0
pytest-mock==3.14.0
pyyaml==6.0.2
requests==2.32.3
sortedcontainers==2.4.0
urllib3==2.3.0
25 changes: 25 additions & 0 deletions .riot/requirements/e49670c.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
# pip-compile --no-annotate .riot/requirements/e49670c.in
#
attrs==24.3.0
certifi==2024.12.14
charset-normalizer==3.4.1
coverage[toml]==7.6.10
dd-trace-api @ git+https://github.com/DataDog/dd-trace-api-py
hypothesis==6.45.0
idna==3.10
iniconfig==2.0.0
mock==5.1.0
opentracing==2.4.0
packaging==24.2
pluggy==1.5.0
pytest==8.3.4
pytest-cov==6.0.0
pytest-mock==3.14.0
pyyaml==6.0.2
requests==2.32.3
sortedcontainers==2.4.0
urllib3==2.3.0
1 change: 1 addition & 0 deletions ddtrace/_monkey.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@
"cassandra": True,
"celery": True,
"consul": True,
"dd_trace_api": True,
"django": True,
"dramatiq": True,
"elasticsearch": True,
Expand Down
Empty file.
79 changes: 79 additions & 0 deletions ddtrace/contrib/internal/dd_trace_api/patch.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
from sys import addaudithook
from typing import Any
from typing import Dict
from typing import List
from typing import Optional
from typing import Tuple
import weakref

import dd_trace_api

import ddtrace


_DD_HOOK_NAME = "dd.hook"
_TRACER_KEY = "Tracer"
_STUB_TO_REAL = weakref.WeakKeyDictionary()
_STUB_TO_REAL[dd_trace_api.tracer] = ddtrace.tracer


def _proxy_span_arguments(args: List, kwargs: Dict) -> Tuple[List, Dict]:
"""Convert all dd_trace_api.Span objects in the args/kwargs collections to their held ddtrace.Span objects"""
proxied_args = []
for arg in args:
if isinstance(arg, dd_trace_api.Span):
proxied_args.append(_STUB_TO_REAL[arg])
else:
proxied_args.append(arg)
proxied_kwargs = {}
for name, kwarg in kwargs.items():
if isinstance(kwarg, dd_trace_api.Span):
proxied_kwargs[name] = _STUB_TO_REAL[kwarg]
else:
proxied_kwargs[name] = kwarg
return proxied_args, proxied_kwargs


def _call_on_real_instance(
operand_stub: dd_trace_api._Stub, method_name: str, retval_from_api: Optional[Any], *args: List, **kwargs: Dict
) -> None:
"""
Call `method_name` on the real object corresponding to `operand_stub` with `args` and `kwargs` as arguments.

Store the value that will be returned from the API call we're in the middle of, for the purpose
of mapping from those Stub objects to their real counterparts.
"""
args, kwargs = _proxy_span_arguments(args, kwargs)
retval_from_impl = getattr(_STUB_TO_REAL[operand_stub], method_name)(*args, **kwargs)
if retval_from_api is not None:
_STUB_TO_REAL[retval_from_api] = retval_from_impl


def _hook(name, hook_args):
"""Called in response to `sys.audit` events"""
if not dd_trace_api.__datadog_patch or name != _DD_HOOK_NAME:
return
args = hook_args[0][0]
api_return_value, stub_self, event_name = args[0:3]
_call_on_real_instance(stub_self, event_name, api_return_value, *args[3:], **hook_args[0][1])


def get_version() -> str:
return getattr(dd_trace_api, "__version__", "")


def patch(tracer=None):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure the integration patch/unpatch approach makes sense here, should we always register an audit hook that listens for these events?

are there ever any cases when we wouldn't want to listen to the events? it would probably be better to have dd_trace_api not emit events if you want to "disable" vs having that handled here.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

patch, unpatch and get_version are here just to make this code work with the existing contrib structure. The contrib is enabled by default, and it can be disabled by the same method as any other default-on contrib. As far as I can tell, this code makes sense as a contrib because that provides lazy loading.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, the lazy loading makes sense, but I am not sure we should provide all the same functionality, like DD_DD_TRACE_API_ENABLED=false or DD_PATCH_MODULES=dd_trace_api:false or patch_all(dd_trace_api=False), etc (or w/e all those options are).

lazy loading, reporting usage and the version to telemetry both definitely make sense.

if getattr(dd_trace_api, "__datadog_patch", False):
return
dd_trace_api.__datadog_patch = True
_STUB_TO_REAL[dd_trace_api.tracer] = tracer
if not getattr(dd_trace_api, "__dd_has_audit_hook", False):
addaudithook(_hook)
emmettbutler marked this conversation as resolved.
Show resolved Hide resolved
dd_trace_api.__dd_has_audit_hook = True


def unpatch():
if not getattr(dd_trace_api, "__datadog_patch", False):
return
dd_trace_api.__datadog_patch = False
# NB sys.addaudithook's cannot be removed
14 changes: 8 additions & 6 deletions lib-injection/sources/min_compatible_versions.csv
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,15 @@ algoliasearch,~=2.5
anthropic,==0.26.0
anyio,>=3.4.0
aredis,0
asgiref,~=3.0
asgiref,~=3.0.0
astunparse,0
async_generator,~=1.10
asyncpg,~=0.23
asyncpg,~=0.23.0
asynctest,==0.13.0
austin-python,~=1.0
avro,0
azure.functions,0
bcrypt,==4.2.1
blinker,0
boto3,==1.34.49
bottle,>=0.12
Expand Down Expand Up @@ -64,19 +65,20 @@ elasticsearch[async],0
envier,==0.5.2
exceptiongroup,0
faiss-cpu,==1.8.0
falcon,~=3.0
falcon,~=3.0.0
fastapi,~=0.64.0
flask,~=0.12.0
flask-caching,~=1.10.0
flask-openapi3,0
gevent,~=20.12.0
git+https://github.com/DataDog/dd-trace-api-py,0
google-ai-generativelanguage,0
google-generativeai,0
googleapis-common-protos,0
graphene,~=3.0.0
graphql-core,~=3.2.0
graphql-relay,0
greenlet,~=1.0.0
greenlet,~=1.0
grpcio,~=1.34.0
gunicorn,==20.0.4
gunicorn[gevent],0
Expand All @@ -97,13 +99,13 @@ langchain-community,==0.0.14
langchain-core,==0.1.52
langchain-openai,==0.1.6
langchain-pinecone,==0.1.0
langchain_experimental,==0.0.47
langgraph,~=0.2.60
logbook,~=1.0.0
loguru,~=0.4.0
lxml,0
lz4,0
mako,~=1.1.0
mariadb,~=1.0.0
mariadb,~=1.0
markupsafe,<2.0
mock,0
molten,>=1.0
Expand Down
14 changes: 8 additions & 6 deletions min_compatible_versions.csv
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,15 @@ algoliasearch,~=2.5
anthropic,==0.26.0
anyio,>=3.4.0
aredis,0
asgiref,~=3.0
asgiref,~=3.0.0
astunparse,0
async_generator,~=1.10
asyncpg,~=0.23
asyncpg,~=0.23.0
asynctest,==0.13.0
austin-python,~=1.0
avro,0
azure.functions,0
bcrypt,==4.2.1
blinker,0
boto3,==1.34.49
bottle,>=0.12
Expand Down Expand Up @@ -64,19 +65,20 @@ elasticsearch[async],0
envier,==0.5.2
exceptiongroup,0
faiss-cpu,==1.8.0
falcon,~=3.0
falcon,~=3.0.0
fastapi,~=0.64.0
flask,~=0.12.0
flask-caching,~=1.10.0
flask-openapi3,0
gevent,~=20.12.0
git+https://github.com/DataDog/dd-trace-api-py,0
google-ai-generativelanguage,0
google-generativeai,0
googleapis-common-protos,0
graphene,~=3.0.0
graphql-core,~=3.2.0
graphql-relay,0
greenlet,~=1.0.0
greenlet,~=1.0
grpcio,~=1.34.0
gunicorn,==20.0.4
gunicorn[gevent],0
Expand All @@ -97,13 +99,13 @@ langchain-community,==0.0.14
langchain-core,==0.1.52
langchain-openai,==0.1.6
langchain-pinecone,==0.1.0
langchain_experimental,==0.0.47
langgraph,~=0.2.60
logbook,~=1.0.0
loguru,~=0.4.0
lxml,0
lz4,0
mako,~=1.1.0
mariadb,~=1.0.0
mariadb,~=1.0
markupsafe,<2.0
mock,0
molten,>=1.0
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
---
features:
- |
dd-trace-api: adds a simple and minimal instrumentation for the dd-trace-api-py package
6 changes: 6 additions & 0 deletions riotfile.py
Original file line number Diff line number Diff line change
Expand Up @@ -756,6 +756,12 @@ def select_pys(min_version: str = MIN_PYTHON_VERSION, max_version: str = MAX_PYT
),
],
),
Venv(
name="dd_trace_api",
command="pytest {cmdargs} tests/contrib/dd_trace_api",
pkgs={"git+https://github.com/DataDog/dd-trace-api-py": latest, "requests": latest},
pys=select_pys(min_version="3.8"),
),
# Django Python version support
# 2.2 3.5, 3.6, 3.7, 3.8 3.9
# 3.2 3.6, 3.7, 3.8, 3.9, 3.10
Expand Down
Empty file.
Loading
Loading