Top Ruby Frameworks & Libraries for web crawling

Quick and dirty web crawling.

Fast high-level web crawling Ruby framework

A loose framework for crawling and scraping web sites.

A Ruby DSL for structured web crawling, with a robust caching system.

ToSBack crawls, archives and tracks changes in terms of service and privacy policies. ToSBack3, inspired by EFF's ToSBack, is built in Ruby on Rails, features a we...

Implements a MongoDB back-end for Spidey (, a framework for crawling and scraping web sites....

A simple Ruby web spider that uses Anemone to crawl every page of a site looking for email addresses. Stores the results with SQLite3 using Data Mapper....

Wgit allows you to crawl and extract the data you want from the web

An example job that converts Common Crawl archived web pages into text

This is a @microverseinc capstone project. It entails building a web-scraper that crawls data about COVID-19 from

A Zillabyte Component for crawling web domains and upload screenshots of webpage contents

A ruby gem for crawling social media stats from various web entities

web crawler to crawl nptel site and extract data

Ruby classes to make it easier to crawl a website, find the items you need (products usually) and save them to CSV...

Crawl web sites and download images/pictures or everything you want

This is a simple web scraper that basically crawls a website and retrieve specific data from the site. Built with Ruby and Nokogiri....

Web crawling with ruby.

Web crawling helper using EM and Hpricot

Simple cached web gem for crawling and caching locally

Web crawling system that parses, screenshots, and indexes web page information. Written in Ruby.

crawls for videos from any given url - just a utility

Crawls, sanitizes and serves web pages in an API

Simple web crawler to crawl a domain and generate sitemap

Build Search Engine Crawlable, Accessible, Deeplinked AJAX Web Apps

A customizable and exploratory fuzz-testing tool written in Ruby. Simulates the web browser to create HTTP requests, parse HTML, and mimic user actions. After craw...

a web crawler developed on ruby on rails platform to crawl medium for particular tags

A code kata... a simple script to crawl a web and download from all Dropbox link

This Web-App crawls student activity data to find free food!

A Rails web application to crawl for scholars citation rankings and present them in table views

Just a practise to crawl geo data from Google Map web services

A simple web scraper built with Ruby and the Nokogiri gem. It crawls a certain website and gets the prices and other data of cryptocurrencies. Rspec was used for t...

Crawls the web for a list of stuff and writes it down in a computer friendly format