GitHub - Vortech-Group/trace: One-way, change-based synchronization of published Eloquent data between two Laravel applications: a source change feed and a remote pull engine.

Sync published Eloquent data from one Laravel application into another, one way, by pulling only what changed.
An explicit field whitelist, a commit-ordered change log and safe local soft deletes.

Why

Copying data between two applications usually ends in a nightly full export, or in a webhook that fires inside a model event and is lost when the request fails. Trace gives the source a small authenticated feed and gives the remote a resumable, idempotent pull.

	Nightly export / model-event webhook	Trace
What leaves the source	whatever the serializer returns	only the fields you whitelist
Change is lost when the call fails	yes	no, it is a durable row in the same transaction
Cursor can skip a slow transaction	with auto-increment ids, yes	no, versions are assigned under a row lock in commit order
Interrupted first import	starts over	resumes from the last committed page
Run twice	duplicates or needless writes	same rows, no `UPDATE` for unchanged values
Record deleted at the source	hard delete, or nothing	local soft delete, only for records Trace linked
Source vanishes or is replaced	quietly overwritten	`instance_id` mismatch is a hard failure
`remote_id` column on your table	usually	never, links live in `trace_links`

By default Trace follows Eloquent model events. Writes that fire none (mass updates, DB::table(), raw SQL, imports) are not logged; a full reconcile repairs them. For lossless tracking, capture changes with database triggers instead. Delivery is at-least-once with idempotent application, never exactly-once.

Requirements

PHP 8.3+
Laravel 12 or 13
On the remote: a cache store with atomic locks (redis, database, file, memcached) and a queue for dispatch()
MySQL 8+, PostgreSQL (SQLite for development and the basic test suite)

Installation

composer require vortech/laravel-trace

The service provider and the Remote facade are registered through package discovery. Then prepare the side(s) the application plays:

php artisan trace:install            # asks which side(s) this application plays
php artisan trace:install --source   # publishes the config and the source migration
php artisan trace:install --remote   # publishes the config and the remote migration
php artisan migrate

Without a flag the command shows a prompt where you pick Source, Remote or both. In a non-interactive run (--no-interaction, CI) a flag is required.

Use --force to overwrite an existing config file. The tags trace-config, trace-source-migrations and trace-remote-migrations are also available through vendor:publish.

Quickstart

Source application

TRACE_SOURCE_TOKEN=long-random-secret

Mark the model. There is no trait and no interface:

use Vortech\Trace\Attributes\Traceable;

#[Traceable]
class Product extends Model
{
    use SoftDeletes;
}

Describe what is published (php artisan trace:make-source ProductSource). $fields is the whitelist: nothing else ever leaves the application, and hidden attributes, passwords, tokens and the primary key are refused.

use Vortech\Trace\Source;

final class ProductSource extends Source
{
    protected string $model = Product::class;

    protected array $fields = ['code', 'name', 'description', 'barcode', 'slug', 'brand', 'category', 'is_active'];
}

Register the feed in routes/api.php:

Route::source('products', ProductSource::class);
// GET /api/products/snapshot  full data, paged, for the first import and reconciles
// GET /api/products/changes   what changed after a cursor

Optionally let the model name its source, then the route can take the model:

#[Traceable(source: ProductSource::class)]
class Product extends Model { use SoftDeletes; }

Route::source('products', Product::class);   // same feed as above

Naming the Source class stays possible, and is the way to publish one model through several feeds with different field lists. A model without source: (or one whose source publishes another model) is refused when the route is registered.

Write through transactions, so a change and its log row commit or roll back together:

DB::transaction(fn () => $product->update(['name' => 'Blue Mug']));

Remote application

TRACE_REMOTE_URL=https://source.example.com/api
TRACE_REMOTE_TOKEN=long-random-secret

Declare the receiving model. SoftDeletes is required, because Trace only ever soft deletes.

use Vortech\Trace\Attributes\Sync;

#[Sync(source: 'products', mapper: ProductMapper::class)]
class Product extends Model
{
    use SoftDeletes;
}

Map the fields (php artisan trace:make-mapper ProductMapper):

use Vortech\Trace\Mapper;

final class ProductMapper extends Mapper
{
    public function fields(): array
    {
        return [
            'name',                // same name on both sides
            'description',
            'barcode' => 'ean',    // remote attribute => local column
            'is_active' => 'active',
        ];
    }

    public function match(): array
    {
        return ['code' => 'product_code'];   // finds an existing local row before it has a link
    }

    public function transform(array $mapped): array   // receives local column names
    {
        return $mapped;
    }
}

Run it:

use Vortech\Trace\Facades\Remote;

$result = Remote::for(Product::class)->sync()->run();                  // SyncResult
Remote::for(Product::class)->sync()->dispatch();                       // queued
Remote::for(Product::class)->sync()->onQueue('integrations')->dispatch();
Remote::for(Product::class)->reconcile()->run();                       // full state comparison

$result->created; $result->updated; $result->unchanged; $result->deleted; $result->restored;
$result->durationMs; $result->cursor;

sync() only describes the operation; run() and dispatch() are the terminal calls. dispatch() returns a PendingDispatch, not a SyncResult.

How it works

The source change log

One synchronous wildcard listener sees created, updated, deleted and restored of every #[Traceable] model (checked by Reflection, cached) and writes a row to trace_changes. It locks the model's trace_streams row (SELECT ... FOR UPDATE), increments version and inserts the change in the same transaction. A higher version therefore cannot commit before a lower one, which is what makes a cursor safe.

The log stores only the key and changed / deleted. The API answers with the current state of the record, or a delete tombstone when it is gone. This is a state-convergence feed, not an event replay: a key may appear more than once in a response and the remote applies it idempotently. A restore is an upsert. A force delete fires deleted first and is logged exactly once.

Stable model names

model_type holds the model's morph class: the class name, or your morph map alias. To keep the log valid when a class is renamed or moved, give the model an alias (on the source with #[Traceable], on the remote with #[Sync]):

#[Traceable(source: ProductSource::class, alias: 'product')]
#[Sync(source: 'products', mapper: ProductMapper::class, alias: 'product')]

Choose it before the first change is logged or the first sync runs: rows stored under another name are not found under the new one (update the model_type columns if you must change it later).

Database capture

Model events miss every write that does not go through a model. Mark the model with CaptureMode::Database and the database writes the change log itself:

#[Traceable(source: ProductSource::class, capture: CaptureMode::Database)]
class Product extends Model { use SoftDeletes; }

php artisan trace:triggers install     # also: remove, status, --sql to print the statements

The triggers do what the listener does (lock the trace_streams row, bump the version, insert the change) inside the transaction of the write, so the log, the commit order, the API and the remote are unchanged. A model captured by triggers is skipped by the listener, so nothing is logged twice. A soft delete is logged as deleted, a restore as changed.

MySQL, MariaDB, PostgreSQL and SQLite; the model must use the connection of the trace tables. Installing needs the TRIGGER privilege (MySQL with binary logging also SUPER or log_bin_trust_function_creators=1); --sql prints the statements for a DBA.
Run install again after you change the table, the key or the soft delete column (trace:doctor reports missing and outdated triggers). One capturing model per table; switching back to CaptureMode::Events needs trace:triggers remove --model=... first.
Every UPDATE statement is logged, even a no-op (harmless to the idempotent remote). TRUNCATE is invisible to triggers: reconcile after it.

The protocol

GET {feed}/snapshot?limit=500&cursor=... returns data[] = {key, attributes} and meta = {protocol, instance_id, feed, schema_hash, watermark, next_cursor, has_more}. Paging is keyset by primary key, never OFFSET. The cursor is opaque, HMAC-signed with app.key, expires, and carries the watermark taken at the start.

GET {feed}/changes?cursor=... (or since=<watermark> for the first call after a snapshot) returns data[] = {version, key, operation: upsert|delete, attributes?}. meta.watermark is the current stream position, and every response carries an X-Trace-Instance header so a remote talking to another installation notices it even on an error.

Situation	Response
Token missing or wrong	`401`
Source token not configured	`503`, never public
HTTP in production	`403`
Unknown feed	`404`
Bad cursor or limit	`422`
Cursor older than the retention, ahead of the source, or an expired snapshot cursor	`410 cursor_expired`

What a run does

Bootstrap (first run). Snapshot pages are applied one transaction each and the page position is stored, so an interrupted import resumes. After the last page the changes since the start watermark are replayed, because a snapshot is not a database snapshot. Only then are linked records that never showed up soft deleted.
Incremental. Pages of /changes. Each page commits its records, links and the cursor together; a failing page leaves the cursor in front of it.
Reconcile (->reconcile(), --full). A complete fresh traversal. It repairs updates and deletes that bypassed model events and soft deletes linked records that vanished.

Per record: find the link; without one, try match() (no hit creates a record, one hit links it, several hits fail with AmbiguousMatchException); copy only the whitelisted fields; save only when something is dirty. A null sent by the source is data, a missing attribute is a schema error and never a silent null. The match columns are set on create only and never touched again.

Safe deletes

Guard	Behaviour
Only linked records are deleted	Local rows Trace did not link are never touched
Soft delete only	The model must use `SoftDeletes` (`trace.safety.require_soft_deletes`)
Partial fetch, timeout, mapping or schema error	No deletion by absence, the cursor stays
Deletion guard	When more than `max_missing_deletion_ratio` (20%) of at least `min_records_for_ratio_guard` links vanish, the run stops with `DeletionGuardException` and `reconcile_required`. Override with `->allowMassDeletion()` or `--allow-mass-deletion`
`instance_id` changed	Hard `InstanceMismatchException`, links of two installations are never mixed
Schema hash changed	`reconcile_required`
Cursor expired	`reconcile_required`, the cursor is never skipped blindly

Queue and locking

TraceSyncJob carries only scalars (model class, mode, flags). One run is active per connection + feed + local model, enforced by an atomic cache lock for run() and the queue alike; a job that finds a running sync leaves the work to it. Timeouts, 429 and 5xx are retried with backoff (trace.remote.retries); 401, 403, schema, mapping and data errors are not. The job timeout is derived from trace.remote.lock_ttl.

Multiple sources

A remote application can pull from more than one source installation. Name the extra sources in config/trace.php and point a model at one of them:

// config/trace.php
'remote' => [
    'default' => ['url' => env('TRACE_REMOTE_URL'), 'token' => env('TRACE_REMOTE_TOKEN')],
    'connections' => [
        'logistics' => ['url' => env('LOGISTICS_TRACE_URL'), 'token' => env('LOGISTICS_TRACE_TOKEN')],
    ],
],

#[Sync(source: 'products', mapper: ProductMapper::class)]                                // the default source
class Product extends Model { use SoftDeletes; }

#[Sync(source: 'warehouses', mapper: WarehouseMapper::class, connection: 'logistics')]   // another installation
class Warehouse extends Model { use SoftDeletes; }

Every model keeps its own state (cursor, bootstrap position, schema hash), its own run lock and its own instance_id check, and links carry the instance_id, so identical keys of two sources never mix. Use as many named connections as you need, Remote::for() and the commands work the same.

One model has one source: #[Sync] is not repeatable and Remote::for() takes no connection. To receive the same table from two installations, declare a model per source.

Artisan

php artisan trace:install [--source] [--remote] [--force]   # no flag: asks which side
php artisan trace:make-source ProductSource
php artisan trace:make-mapper ProductMapper
php artisan trace:sync 'App\Models\Product'
php artisan trace:sync 'App\Models\Product' --queue
php artisan trace:sync 'App\Models\Product' --full [--allow-mass-deletion]
php artisan trace:sync 'App\Models\Product' --dry-run [--full]   # rehearse, write nothing
php artisan trace:triggers [install|remove|status] [--model=] [--sql]
php artisan trace:doctor [--offline]
php artisan trace:prune

trace:doctor checks the token, tables, source and mapper declarations, the capture triggers, columns, connections, the lock-capable cache, the sync state and, unless --offline, the source itself (protocol, instance_id, schema hash).

Dashboard

A small dark dashboard for both sides: the synced models (status, cursor, recent runs), the whole run history, the published feeds (capture mode, triggers, changes per hour, retention) and the trace:doctor checks. It is off by default:

TRACE_DASHBOARD_ENABLED=true

It lives under /trace (trace.dashboard.path and .middleware) and answers 404 everywhere but the local environment until you define the gates, for example in AppServiceProvider:

Gate::define('viewTrace', fn (?User $user) => $user?->isAdmin() === true);
Gate::define('manageTrace', fn (?User $user) => $user?->isAdmin() === true);

viewTrace opens it; manageTrace also allows Sync now, Reconcile, a dry run, Prune now and Install triggers. A sync queues the usual job, unless you tick Run now, without the queue (fine for a quick check, a long bootstrap belongs on the queue).

Dry run

Before the first sync of a table that already holds data, rehearse it (also Remote::for(...)->sync()->dryRun()->run() and a checkbox on the dashboard):

php artisan trace:sync 'App\Models\Product' --dry-run

The source is read and every record goes through the real code in a transaction that is rolled back. You get how many records would be created, matched to an existing row through match(), updated, deleted or restored, and the error a real run would hit (an ambiguous match, a constraint). Nothing is written, no event fires, the state does not move.

Without a match() in the mapper existing rows are never matched and every record becomes a new row (the command warns). A dry run reads the whole feed in one transaction, so try a very large table off peak hours, and it cannot be queued.

Scheduling

use Illuminate\Support\Facades\Schedule;

Schedule::call(fn () => Remote::for(Product::class)->sync()->dispatch())->everyFiveMinutes();
Schedule::call(fn () => Remote::for(Product::class)->reconcile()->dispatch())->weekly();
Schedule::command('trace:prune')->daily();   // on the source

Configuration

config/trace.php:

Key	Purpose
`connection`	Connection of the `trace_*` tables. It must be the connection of the traceable and synced models
`source.token`	Bearer token the remote sends. Empty keeps the feed closed (`503`)
`source.page_size`, `max_page_size`	Default and maximum page size
`source.retention_days`	Change log retention, 30 by default
`source.rate_limit`	Requests per minute and IP, `0` disables it
`source.snapshot_cursor_ttl`	Minutes a snapshot cursor stays valid
`source.forbidden_fields`	Fields no Source may publish
`remote.default`	`url` and `token` of the default source
`remote.connections`	Named sources for `#[Sync(connection: 'logistics')]`, see Multiple sources
`remote.models`	Extra models for `trace:doctor`
`remote.timeout`, `retries`, `queue`, `lock_ttl`, `runs_retention_days`	HTTP and queue behaviour
`dashboard`	`enabled`, `path` and `middleware` of the dashboard
`safety`	`require_soft_deletes`, `max_missing_deletion_ratio`, `min_records_for_ratio_guard`

The source signs its cursors with app.key, so that key must be set there. remote.default.url is the base URL of the source's API prefix; Trace appends the feed and the endpoint. Production requires HTTPS.

Events

SyncStarted, SyncCompleted and SyncFailed carry the TraceSyncRun (and the SyncResult or the exception). The logs trace.sync.completed and trace.sync.failed contain counters and identifiers, never record values.

Good to know

Source write	Tracked
Eloquent model write inside `DB::transaction()`	Atomic: the change and its log row commit or roll back together
Eloquent model write without a transaction	Best effort, with a small crash window between the write and the log. A failed log write is reported, not hidden
`Model::where()->update()`, bulk `delete()`, `DB::table()`, raw SQL, imports	Not logged with model events, the next reconcile repairs them. Logged atomically with `CaptureMode::Database`

Reliable tracking needs a model event, lossless tracking needs a transaction. A full reconcile repairs missed events. Use CaptureMode::Database when writes bypass models; binlog/WAL based CDC is not part of Trace.
One connection. The trace tables must live on the connection of the traceable models, and on the remote of the synced models. Anything else fails fast.
Without match() and without a link Trace creates a new row. On a table that already holds the data this duplicates it, so define match().
Retention. Keep retention_days longer than your longest bootstrap and the longest pause between syncs.
Deploys. The listener also logs writes from queue workers, CLI and cron. Restart long-running workers on the source after a deployment.
Page size and memory. A page is applied in one transaction, keep page_size reasonable for wide tables.

Testing

composer test
composer analyse
composer lint

The suite uses Pest and runs on SQLite through Orchestra Testbench; the remote talks to the source of the same application through the real HTTP kernel. Static analysis runs at PHPStan level max with the strict rules.

Row locks need a real server. tests/Concurrency is skipped unless you point it at MySQL or PostgreSQL (the CI runs both):

TRACE_TEST_DB=mysql TRACE_TEST_USERNAME=root TRACE_TEST_PASSWORD=secret TRACE_TEST_DATABASE=trace_test vendor/bin/pest tests/Concurrency

Changelog

See CHANGELOG for what has changed recently.

Contributing

See CONTRIBUTING for details.

Security

If you discover a security issue, please slug mate@vortech.hu instead of using the issue tracker.

Credits

Mate Papp, Developer @ Vortech

License

The MIT License (MIT). See the License File for more information.

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
art		art
config		config
database/migrations		database/migrations
dist		dist
resources		resources
routes		routes
src		src
stubs		stubs
tests		tests
.gitattributes		.gitattributes
.gitignore		.gitignore
.nvmrc		.nvmrc
CHANGELOG.md		CHANGELOG.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE.md		LICENSE.md
README.md		README.md
composer.json		composer.json
package-lock.json		package-lock.json
package.json		package.json
phpstan.neon.dist		phpstan.neon.dist
phpunit.xml		phpunit.xml
pint.json		pint.json
tsconfig.json		tsconfig.json
vite.config.ts		vite.config.ts

Navigation Menu

Repository navigation

About

Repository files navigation

Why

Requirements

Installation

Quickstart

Source application

Remote application

How it works

The source change log

Stable model names

Database capture

The protocol

What a run does

Safe deletes

Queue and locking

Multiple sources

Artisan

Dashboard

Dry run

Scheduling

Configuration

Events

Good to know

Testing

Changelog

Contributing

Security

Credits

License

About

Releases

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

Why

Requirements

Installation

Quickstart

Source application

Remote application

How it works

The source change log

Stable model names

Database capture

The protocol

What a run does

Safe deletes

Queue and locking

Multiple sources

Artisan

Dashboard

Dry run

Scheduling

Configuration

Events

Good to know

Testing

Changelog

Contributing

Security

Credits

License

About

Topics

Resources

Contributing

Stars

Watchers

Forks

Releases

Contributors

Languages