Repository navigation
Add PCAP import module for files, email and network indicators - #1010
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e178209652
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| else: | ||
| urls = re.findall( | ||
| r"https?://(?:[-\w.]|(?:%[\da-fA-F]{2}))+", | ||
| r"""https?://[^\s<>"']+""", |
There was a problem hiding this comment.
The broader pattern now matches common defanged or malformed text such as http://example[.]com and http://:, but URLObject can reject these with MISPObjectException rather than the currently caught ValueError. In an email containing such text, URL extraction therefore aborts the entire import and the module server returns its generic failure response instead of skipping the invalid URL; catch the PyMISP parser exception or validate these broader matches first.
Useful? React with 👍 / 👎.
| if report.stat().st_size > MAX_OUTPUT_SIZE: | ||
| raise ValueError("The capture's JSON metadata exceeds the 100 MiB limit.") |
There was a problem hiding this comment.
For a capture containing many small packets or highly compressed HTTP content, TShark can write far beyond 100 MiB to packets.json and the export directories before this check runs. TShark documents that --export-objects saves every object directly into the destination directory, so the later size checks do not actually protect the service filesystem from exhaustion; monitor and terminate the subprocess at the configured limit or run it under an enforced storage quota.
Useful? React with 👍 / 👎.
Closes #397.
Adds
pcap_importto import PCAP and PCAPNG captures into MISP. TShark handles offline decoding and TCP reassembly, including out-of-order segments and retransmissions. The module returns IPv4/IPv6 endpoints, DNS names and answers, HTTP URLs/hosts/user agents, and TLS SNI hostnames. Observed network attributes haveto_idsdisabled for analyst review.HTTP payloads become file objects with hashes and data. Reassembled SMTP messages reuse
email_importto produce email objects, attachments and body URLs; malformed messages are preserved as files. Also fixes the email importer's plain-text URL matching to retain paths, queries and fragments, while skipping malformed URLs that raise PyMISP parser exceptions.Requires TShark >= 4.0 and PyMISP's email/fileobjects extras. Payload extraction requires complete, unencrypted HTTP/SMTP traffic; HTTPS provides visible TLS metadata only. Includes capture-size validation, configurable subprocess timeout, temporary-file cleanup, module documentation, and TShark installation in CI. JSON metadata, combined HTTP/SMTP exports, and diagnostic output are monitored every 50 ms while TShark runs; exceeding a limit kills and reaps the process. Checks also run after exit. Bursts can overshoot between polls, so this monitoring is not a filesystem quota.
Validation:
/modulesand successful capture import through/query.