Page 2 - 记录 - Gary's Blog

Selenium infinite scrolling to bottom using python

2023-09-19 232 Crawler, Python, Selenium Leave a Comment

We have 3 kinds of infinite scrolling to bottom while using selenium. Simply scrolling - Stop until there is no more content. Meet scrolling - Stop until we meet a speciy element. Partial scrolling - Scrolling content in specify html container, stop until there are no more content. All these 3 scenses will use …

How to install OS and connect to NanoPi-R2S (rk3328)

2023-09-12 301 Leave a Comment

Official wiki page OS image on Google Drive Install OS Go to google drive, navigate to: 01_Official images/01_SD card images. Download any image, e.g., "rk3328-sd-debian-bullseye-core-5.15-arm64-20230529.img.gz". Unzip it to "rk3328-sd-debian-bullseye-core-5.15-arm64-20230529.img". importat…

The most commonly used functions in machine learning data cleaning

2023-09-05 58 Data Cleaning, Machine Learning, ML Leave a Comment

1. Get null percentage of columns def get_null_percentage(df): count = df.isnull().sum() percent = (df.isnull().mean() * 100).round(2) summary = pd.DataFrame({'count': count, 'percentage': percent}) return summary[summary["count"] > 0].sort_values(ascending=False, by="perce…

How to find all indicators and country codes while using imf api

2023-08-08 321 Leave a Comment

When following the imf api service to query data, you have to specify the indicators. You must have viewed this 2 posts: How to Use the API (Python and R) IMF API (Python) But both of them do not explain how to get full list of indicators as well as the country code. Get full list of imf indicators We have to a…

What is OpenAI Function Calling and How to Use it?

2023-06-17 178 ChatGPT, OpenAI 1 Comment

Simply put, OpenAI / ChatGPT function calling can identify the intent and entities in a sentence through a language model. For example, when the user asks "I want to recharge my phone100 USD". The model should recognize the intent recharge and it's 2 entities recharge amont: 100 currency:USD Back to O…

A script to test MySQL port is running

2023-05-03 95 MySQL Leave a Comment

#!/bin/bash ip="<MySQL HOST>" port="<MySQL PORT>" output_file="<Log File Path>" while : do nc -zv "$ip" "$port" > /dev/null datetime=$(date +"%Y-%m-%d %H:%M:%S") if [ $? -eq 0 ]; then echo "$datetime: Port $port is open on $ip…

From PHP to Python In 10 Minutes

2023-04-11 132 PHP, Python Leave a Comment

PHP and Python are both high-level, interpreted programming languages with a variety of applications, but they have some significant differences in terms of syntax, design philosophy, and use cases. Here's a detailed comparison of the two languages: 1. Purpose and Use Cases PHP: Created by Rasmus Lerdorf in 199…

How to compress mp4 in one command with ffmpeg

2023-03-28 94 ffmpeg, mp4 Leave a Comment

With ffmpeg, we can create a free video compresser on our local machine. First, create a shell function on you .bashrc or .zshrc: function mp4-compress() { filename=$(basename -- "$1") extension="${filename##*.}" name="${filename%.*}" ffmpeg -i "$1" -vcodec libx265 -crf 2…

How to transfer data between different mongodb instance?

2023-03-25 68 MongoDB Leave a Comment

Here is the python code to transfer data between different mongo instance: def transfer(source_collection, dest_collection, batch_size: int): offset = 0 total = source_collection.count_documents({}) total_inserted = 0 while True: documents = source_collection.find().skip(offset).limit(batch_size) data = list(do…

How to config logging output to file and stdio/console with basicConfig() in one line?

2023-03-22 109 Python Leave a Comment

import logging logging.basicConfig(format='[%(asctime)s] [%(levelname)s] %(message)s', datefmt='%Y-%d-%m %I:%M:%S', level=logging.DEBUG, handlers=[logging.FileHandler('app.log'), logging.StreamHandler()])