Skip to content
radkissonPublic

About

Docker-based headless Selenium automation toolkit for web scraping and form submission tasks. Features cross-platform setup scripts, configurable workflows via INI files, and a containerized Chrome environment for reliable, reproducible automation.

Topics

Resources

Security policy

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

automads

A Docker-based automation tool for posting classified ads using Selenium 4 with headless Chrome. Now featuring improved setup, configuration management, and cross-platform support for easier development and deployment.

πŸš€ Quick Start

Prerequisites

One-Command Setup

Linux/macOS:

./setup.sh

Windows:

setup.bat

Development setup:

./setup.sh --dev  # Includes testing and development tools

Quick Run

With Docker (Recommended):

make run
# or
docker compose up -d selenium && docker compose run --rm python

Local Development:

make run-local
# or
source venv/bin/activate && python main.py

Verify Setup:

make verify        # Check setup completeness
make health-check  # Test Selenium connectivity

πŸ“‹ Available Commands

We've added a Makefile for easy command management:

make help          # Show all available commands
make setup         # Set up the project
make setup-dev     # Set up for development
make verify        # Verify setup is working
make health-check  # Check system health
make run           # Run with Docker
make run-local     # Run locally
make test          # Run tests
make format        # Format code
make lint          # Run linting
make clean         # Clean up generated files

βš™οΈ Configuration

Environment Variables

Create a .env file (automatically created from .env.example during setup):

# Required credentials
UDEC_USERNAME=your_username_here
UDEC_PASSWORD=your_password_here

# Optional configuration
SELENIUM_URL=http://selenium:4444/wd/hub
HEADLESS=true
WAIT_TIMEOUT=10
LOG_LEVEL=INFO

Advanced Configuration

Copy config.ini.example to config.ini and customize the automation behavior:

  • Form data (title, description, phone, location)
  • User ID in the create_ad_url
  • Selenium timeouts and options
  • URLs and endpoints

Important: Never commit your actual config.ini file as it contains personal information.

πŸ› οΈ Development

VS Code Support

This project includes full VS Code support:

  1. DevContainer: Open in VS Code and use "Reopen in Container" for a complete development environment
  2. Extensions: Recommended extensions are automatically suggested
  3. Settings: Pre-configured Python, linting, and testing settings

Local Development Setup

# Set up development environment
./setup.sh --dev

# Activate virtual environment
source venv/bin/activate  # Linux/macOS
# or
venv\Scripts\activate     # Windows

# Run tests
make test
# or
python -m pytest

# Format code
make format
# or
black .

# Lint code
make lint
# or
flake8 *.py

Running Tests

# Run all tests
make test

# Run tests with coverage
python -m pytest --cov=. --cov-report=html

# Run specific test file
python -m pytest tests/test_main.py -v

🐳 Docker Usage

With Docker Compose (Recommended)

# Start all services
docker compose up -d

# Run automation
docker compose run --rm python

# View Selenium in browser (for debugging)
# Visit http://localhost:7900 (password: secret)

# Stop services
docker compose down

Manual Docker Commands

# Build images
docker compose build

# Start only Selenium
docker compose up -d selenium

# Run automation manually
docker compose run --rm python python main.py

πŸ”§ Features

Improved Error Handling

  • Comprehensive logging to both console and file
  • Better error messages with troubleshooting hints
  • Environment validation before running

Cross-Platform Support

  • Works on Windows, macOS, and Linux
  • Platform-specific setup scripts
  • Virtual environment automation

Development Tools

  • Pre-commit hooks and code formatting
  • Testing infrastructure with pytest
  • Type checking with mypy
  • Linting with flake8

Configuration Management

  • Environment variable validation
  • .env file support with examples
  • Configurable automation parameters
  • Separate development dependencies

πŸ“Š Monitoring & Debugging

Logs

  • Console output with colored logging levels
  • File logging to automation.log
  • Configurable log levels (DEBUG, INFO, WARNING, ERROR)

Selenium Debugging

  • VNC access on port 7900 for visual debugging
  • Health checks for service monitoring
  • Better Chrome options for stability

Testing

# Run all tests
make test

# Run with coverage report
python -m pytest --cov=. --cov-report=html
open htmlcov/index.html  # View coverage report

🚨 Troubleshooting

Common Issues

"Docker not found"

  • Install Docker Desktop from docker.com
  • Make sure Docker service is running

"Python not found"

  • Install Python 3.8+ from python.org
  • Make sure Python is in your PATH

"Missing environment variables"

  • Run setup script: ./setup.sh
  • Edit .env file with your credentials
  • Check .env.example for reference

"Selenium connection failed"

  • Start Selenium: docker compose up -d selenium
  • Wait 30 seconds for Selenium to be ready
  • Check Docker logs: docker compose logs selenium

"Permission denied on setup.sh"

chmod +x setup.sh
./setup.sh

Debug Mode

Set environment variables for debugging:

export LOG_LEVEL=DEBUG
export HEADLESS=false  # See browser actions

πŸ”’ Security

  • Environment variables are not committed to git
  • .env files are automatically ignored
  • config.ini is ignored to prevent committing personal information
  • Always use .env.example and config.ini.example as templates
  • Credentials and personal data are only stored locally
  • Docker containers run with minimal privileges

Important: First-Time Setup

If you're setting up this project for the first time:

  1. Copy config.ini.example to config.ini
  2. Copy .env.example to .env
  3. Edit both files with your actual credentials and information
  4. Never commit these files to version control

Note: Previous versions of this repository may have contained example data in config files. Always use the template files and never commit sensitive information.

🀝 Contributing

  1. Fork the repository
  2. Set up development environment: ./setup.sh --dev
  3. Create a feature branch
  4. Make your changes
  5. Run tests: make test
  6. Format code: make format
  7. Submit a pull request

πŸ“œ License

This is free and unencumbered software released into the public domain. See LICENSE for details.

πŸ†˜ Support

For issues and questions:

  1. Check the troubleshooting section above
  2. Review logs in automation.log
  3. Open an issue on GitHub with full error details

About

Docker-based headless Selenium automation toolkit for web scraping and form submission tasks. Features cross-platform setup scripts, configurable workflows via INI files, and a containerized Chrome environment for reliable, reproducible automation.

Topics

Resources

Security policy

Stars

1 star

Watchers

0 watching

Forks

Contributors

Languages