zinc cobalt
This project can turn a YAML file into a application at runtime. The YAML file is a config for a application think of it like a application spec. Instead of writing code you edit a YAML file and the application will generate a full application for that YAML file thus saving the developer time coding it themselves. (It does not use AI, fully dynamically coded)
It currently supports multiple different dynamic modules for example a database, HTTP, OAuth, polling, entry points (web/disc bot).
Instead of structuring the HTTP request and parsing the html or json it does it all for you. Instead of writing the data access layer logic for a database it can do CRUD tasks for you. It can parse Swagger API docs, parse http responses (handles nested objects and nested arrays). I've recently added support for HTTP API scanning allowing users to enter a website URL it will automatically scan for an API and will do the rest.
Currently the only focus of this project is to automate web scraping projects and turn them into full commercial projects within minutes of configuration.
Turn any web scraping idea into a full setup application which can scrape most if not any website, api doc, etc.
Just wanted to share my current progress, any suggestions?
Quick little example for a Minecraft scraper:
websites:
liteban:
- https://bans.fadecloud.com/bans.php
- https://stoneworks.gg/bans/bans.php
commands:
LitebanScraper:
type: http
inputs: [page]
outputs: [player, minecraft_uuid, reason, minecraft_username]
save_to_db: true
multiple_results: true
polling_info:
interval: 0.1
input_type: increasing_page
database_info:
name: mongodb
url: mongodb://localhost:27017/
key_mapper:
uuid: minecraft_uuid
player: minecraft_username
players_name: minecraft_username
players_server: minecraft_server
name: minecraft_username
playerName: minecraft_username
fervent knot
winter ferry
grave ingot