SQL REFACTOR

take flag and name of chat messages from user tabeltake flag and name of chat messages from user tabel
alter user tabel
add lastIp to User
add Corse-Control-Max-Age header to limit preflights
adjust proxycheck
Cache disposable mail providers
Check if Burst tokan is active from API
save lastIp of user
broadcast IPInfo of user on shards
add UserIP join table
rename isAllowed to ipUserIntel
remove unused trash
add userlvl column to RegUser which replaces verified and roles
Make userlvl a tinyint
move third party IDs into own table
show empty line for private users in rankings, closes #38
debug whois
make uuid binary(16)
update canvas center picture
replace XFromRequest functions with express middleware added getters
add MySQL functions for converting IP string to/from varbinary
parse whois ranges from all various formats into numbers
save whois info to ranges into own table and write sql proocedures
change Math.log2 logic in scale calculations
This commit is contained in:
HF 2022-10-19 11:32:43 +02:00
parent d5469f7dc6
commit 00cbddd6c9
77 changed files with 1885 additions and 1039 deletions

View File

@ -24,7 +24,7 @@ Click or tab: Place Pixel
## Build
### Requirements
- [nodejs environment](https://nodejs.org/en/) (>=16)
- [nodejs environment](https://nodejs.org/en/) (>=18)
- Linux or WSL if you want to be safe (we do not build on Windows and therefor can't guarantee that it will work there)
### Building
@ -57,7 +57,7 @@ git config --global url.https://github.com/.insteadOf git://github.com/
## Run
### Requirements
- nodejs environment with [npm](https://www.npmjs.com/get-npm) (>=16)
- nodejs environment with [npm](https://www.npmjs.com/get-npm) (>=18)
- [pm2](https://github.com/Unitech/pm2) (`npm install -g pm2`) as process manager and for logging
- [redis](https://redis.io/) as database for storìng the canvas
- mysql or mariadb ([setup own user](https://www.digitalocean.com/community/tutorials/how-to-create-a-new-user-and-grant-permissions-in-mysql) and [create database](https://www.w3schools.com/SQl/sql_create_db.asp) for pixelplanet) for storing additional data like IP blacklist

57
SQLMIGRATION-TODO.md Normal file
View File

@ -0,0 +1,57 @@
# SQL Migration
User Table:
verified -> if != 0 -> userlvl = 20
roles -> if 1 -> userlvl = 100
delete verified
delete roles
check threepid associations and all logins
discordid -> if != NULL -> ThreePIDs add uid, discordid, provider = 1
redditid -> if != NULL -> ThreePIDs add uid, discordid, provider = 2
delete discordid
delete redditid
# rwhois
Error on WHOIS 69.178.112.123 rwhois.gci.net:4321: no rwhois support
Error on WHOIS 50.7.93.84 rwhois.fdcservers.net:4321: no rwhois support
Error on WHOIS 2605:a601:a904:cc00:5918:f4cd:1dd1:a9c rwhois.googlefiber.net:8987: getaddrinfo ENOTFOUND rwhois.googlefiber.net:8987
Error on WHOIS 198.16.70.52 rwhois.fdcservers.net:4321: no rwhois support
Error on WHOIS 198.16.78.45 rwhois.fdcservers.net:4321: no rwhois support
Error on WHOIS 149.34.217.96 rwhois.cogentco.com:4321: no rwhois support
Error on WHOIS 38.9.254.98 rwhois.cogentco.com:4321: no rwhois support
Error on WHOIS 38.54.79.203 rwhois.cogentco.com:4321: no rwhois support
Error on WHOIS 38.91.101.217 rwhois.cogentco.com:4321: no rwhois support
Error on WHOIS 38.10.69.98 rwhois.cogentco.com:4321: no rwhois support
Error on WHOIS 50.7.93.28 rwhois.fdcservers.net:4321: no rwhois support
Error on WHOIS 38.128.66.211 rwhois.cogentco.com:4321: no rwhois support
Error on WHOIS 198.16.66.125 rwhois.fdcservers.net:4321: no rwhois support
Error on WHOIS 198.16.74.44 rwhois.fdcservers.net:4321: no rwhois support
Error on WHOIS 198.16.70.28 rwhois.fdcservers.net:4321: no rwhois support
Error on WHOIS 198.16.66.197 rwhois.fdcservers.net:4321: no rwhois support
Error on WHOIS 38.54.57.78 rwhois.cogentco.com:4321: no rwhois support
Error on WHOIS 198.16.66.195 rwhois.fdcservers.net:4321: no rwhois support
Error on WHOIS 198.16.66.101 rwhois.fdcservers.net:4321: no rwhois support
Error on WHOIS 2604:3d09:217e:96e0:e121:2b82:29b7:3fa9 rwhois.shawcable.net:4321: no rwhois support
Error on WHOIS 149.57.29.157 rwhois.cogentco.com:4321: no rwhois support
Error on WHOIS 38.91.100.42 rwhois.cogentco.com:4321: no rwhois support
Error on WHOIS 149.71.172.36 rwhois.cogentco.com:4321: no rwhois support
Error on WHOIS 149.36.50.199 rwhois.cogentco.com:4321: no rwhois support
Error on WHOIS 50.5.243.186 rwhois.fuse.net:4321: no rwhois support
Error on WHOIS 65.78.19.109 rwhois.rcn.net:4321: no rwhois support
Error on WHOIS 160.2.105.243 rwhois.cableone.net:4321: no rwhois support
Error on WHOIS 50.7.142.179 rwhois.fdcservers.net:4321: no rwhois support
Error on WHOIS 198.16.66.140 rwhois.fdcservers.net:4321: no rwhois support
Error on WHOIS 149.100.25.120 rwhois.cogentco.com:4321: no rwhois support
Error on WHOIS 50.7.93.28 rwhois.fdcservers.net:4321: no rwhois support
Error on WHOIS 38.34.185.140 rwhois.cogentco.com:4321: no rwhois support
Error on WHOIS 198.16.78.44 rwhois.fdcservers.net:4321: no rwhois support
Error on WHOIS 198.16.66.155 rwhois.fdcservers.net:4321: no rwhois support
Error on WHOIS 38.107.255.226 rwhois.cogentco.com:4321: no rwhois support
Error on WHOIS 184.155.140.22 rwhois.cableone.net:4321: no rwhois support
2600:1001:b00c:7158:0000:0000:0000:0000
181.165.235.69

View File

@ -16,6 +16,9 @@ Pixelplanet has its own git repository for deployment on the live system, if an
## rebuild.sh
script to manually trigger rebuilding and restarting pixelplanet on the server
## banlist.csv
CIDR banlist for firewall
## Some notes:
Cloudflare Caching Setting `Broser Cache Expiration` should be set to `Respect Existing Headers` or it would default to 4h, which is unreasonable for chunks.
Additinally make sure that cachebreakers get blocked by setting Cloudflare Firewall rules to block empty query strings at least for chunks

103
deployment/banlist.csv Normal file
View File

@ -0,0 +1,103 @@
102.129.143.86,list
107.175.73.54,list
107.179.20.204,list
109.175.107.110,list
109.95.217.242,list
114.232.194.255,list
115.226.153.176,list
116.202.22.43,list
119.74.167.169,list
122.252.253.100,list
131.153.26.188,list
131.153.26.189,list
138.3.244.208,list
142.44.128.0/17,OVH
146.70.52.46,list
154.16.192.176,list
172.107.240.26,list
176.115.102.106,list
176.215.80.25,list
176.36.75.28,list
178.163.117.211,list
178.20.142.170,list
178.72.81.231,list
181.119.30.41,list
185.177.124.224,list
185.26.63.16,list
188.132.139.129,list
188.3.187.228,list
190.11.138.163,list
190.49.124.184,list
191.101.31.95,list
191.96.150.204,list
192.142.16.11,list
194.167.0.0/16,frenchy university
194.182.67.126,list
194.87.239.144,list
212.237.228.220,list
212.237.228.225,list
213.227.154.74,list
2400:8901::/64,captcha flood
2600:3c03::/64,captcha flood
2600:3c04::/64,captcha flood
2607:5300:203:14ae::/64,list
2a01:4ff:f0:3004::/64,websocket flood
2a02:27b0:4d03:c9d0::/64,websocket flood
31.210.107.199,list
34.245.86.143,list
34.66.88.30,list
34.71.79.206,list
34.82.153.53,list
37.188.11.119,list
37.212.64.134,list
45.189.115.205,list
46.246.41.153,list
46.246.41.158,list
51.161.0.0/17,ovh
51.161.128.0/17,ovh
51.178.0.0/16,ovh
5.142.42.184,list
5.173.137.17,list
5.173.172.228,list
52.143.155.67,list
52.205.26.222,list
5.227.29.33,list
5.227.31.1,list
62.244.51.28,list
77.47.170.231,list
78.132.55.22,list
78.36.107.187,list
78.86.5.152,list
79.45.161.163,list
80.169.156.52,list
81.92.203.83,list
82.193.110.12,list
84.124.251.104,list
84.17.43.24,list
84.17.56.184,list
84.239.40.225,list
85.76.77.85,list
86.138.152.227,list
86.186.250.150,list
87.7.200.139,list
89.108.99.115,list
89.40.143.192,list
91.121.210.56,list
91.228.236.175,list
92.32.69.242,list
93.115.28.181,list
95.181.236.133,list
45.9.88.0/22,ddos230612
75.127.7.192/27,ddos230612
85.98.55.199,sch bot
2a01:cb04:60f:2900::/64,sch bot
67.255.77.34,sch bot
188.23.51.246,sch bot
78.174.247.162,sch bot
95.76.46.200,sch bot
88.231.207.77,sch bot
151.135.28.236,sch bot
2a0d:6fc2:54c0:7500::/64,ws ddos
176.231.133.107,ws ddos
80.121.26.216,sch bot
88.237.43.71,sch bot
1 102.129.143.86 list
2 107.175.73.54 list
3 107.179.20.204 list
4 109.175.107.110 list
5 109.95.217.242 list
6 114.232.194.255 list
7 115.226.153.176 list
8 116.202.22.43 list
9 119.74.167.169 list
10 122.252.253.100 list
11 131.153.26.188 list
12 131.153.26.189 list
13 138.3.244.208 list
14 142.44.128.0/17 OVH
15 146.70.52.46 list
16 154.16.192.176 list
17 172.107.240.26 list
18 176.115.102.106 list
19 176.215.80.25 list
20 176.36.75.28 list
21 178.163.117.211 list
22 178.20.142.170 list
23 178.72.81.231 list
24 181.119.30.41 list
25 185.177.124.224 list
26 185.26.63.16 list
27 188.132.139.129 list
28 188.3.187.228 list
29 190.11.138.163 list
30 190.49.124.184 list
31 191.101.31.95 list
32 191.96.150.204 list
33 192.142.16.11 list
34 194.167.0.0/16 frenchy university
35 194.182.67.126 list
36 194.87.239.144 list
37 212.237.228.220 list
38 212.237.228.225 list
39 213.227.154.74 list
40 2400:8901::/64 captcha flood
41 2600:3c03::/64 captcha flood
42 2600:3c04::/64 captcha flood
43 2607:5300:203:14ae::/64 list
44 2a01:4ff:f0:3004::/64 websocket flood
45 2a02:27b0:4d03:c9d0::/64 websocket flood
46 31.210.107.199 list
47 34.245.86.143 list
48 34.66.88.30 list
49 34.71.79.206 list
50 34.82.153.53 list
51 37.188.11.119 list
52 37.212.64.134 list
53 45.189.115.205 list
54 46.246.41.153 list
55 46.246.41.158 list
56 51.161.0.0/17 ovh
57 51.161.128.0/17 ovh
58 51.178.0.0/16 ovh
59 5.142.42.184 list
60 5.173.137.17 list
61 5.173.172.228 list
62 52.143.155.67 list
63 52.205.26.222 list
64 5.227.29.33 list
65 5.227.31.1 list
66 62.244.51.28 list
67 77.47.170.231 list
68 78.132.55.22 list
69 78.36.107.187 list
70 78.86.5.152 list
71 79.45.161.163 list
72 80.169.156.52 list
73 81.92.203.83 list
74 82.193.110.12 list
75 84.124.251.104 list
76 84.17.43.24 list
77 84.17.56.184 list
78 84.239.40.225 list
79 85.76.77.85 list
80 86.138.152.227 list
81 86.186.250.150 list
82 87.7.200.139 list
83 89.108.99.115 list
84 89.40.143.192 list
85 91.121.210.56 list
86 91.228.236.175 list
87 92.32.69.242 list
88 93.115.28.181 list
89 95.181.236.133 list
90 45.9.88.0/22 ddos230612
91 75.127.7.192/27 ddos230612
92 85.98.55.199 sch bot
93 2a01:cb04:60f:2900::/64 sch bot
94 67.255.77.34 sch bot
95 188.23.51.246 sch bot
96 78.174.247.162 sch bot
97 95.76.46.200 sch bot
98 88.231.207.77 sch bot
99 151.135.28.236 sch bot
100 2a0d:6fc2:54c0:7500::/64 ws ddos
101 176.231.133.107 ws ddos
102 80.121.26.216 sch bot
103 88.237.43.71 sch bot

View File

@ -1,6 +1,6 @@
#!/bin/bash
# This hook builds pixelplanet after a push, and deploys it, it should be ron post-receive
# If it is the master branch, it will deploy it on the life system, and other branch will get deployed to the dev-canvas (a second canvas that is running on the server)
# This hook builds pixelplanet after a push, and deploys it, it should be run post-receive
# If it is devel or test branch, it will deploy it on the test system.
#
# To set up a server to use this, you have to go through the building steps manually first.
#
@ -8,8 +8,6 @@
BUILDDIR="/home/pixelpla/pixelplanet-build"
#folder for dev canvas
DEVFOLDER="/home/pixelpla/pixelplanet-dev"
#folder for production canvas
PFOLDER="/home/pixelpla/pixelplanet"
should_reinstall () {
local TMPFILE="${BUILDDIR}/package.json.${1}.tmp"
@ -53,28 +51,10 @@ do
GIT_WORK_TREE="$BUILDDIR" GIT_DIR="${BUILDDIR}/.git" git fetch --all
cd "$BUILDDIR"
branch=$(git rev-parse --symbolic --abbrev-ref $refname)
if [ "master" == "$branch" ]; then
echo "---UPDATING REPO ON PRODUCTION SERVER---"
GIT_WORK_TREE="$BUILDDIR" GIT_DIR="${BUILDDIR}/.git" git reset --hard "origin/$branch"
COMMITS=`git log --pretty=format:'- %s%b' $newrev ^$oldrev`
COMMITS=`echo "$COMMITS" | sed ':a;N;$!ba;s/\n/\\\n/g'`
echo "---BUILDING pixelplanet---"
should_reinstall master
DO_REINSTALL=$?
[ $DO_REINSTALL -eq 0 ] && npm_reinstall
npm run build
echo "---RESTARTING CANVAS---"
pm2 stop ppfun-server
pm2 stop ppfun-backups
copy "${PFOLDER}" "${DO_REINSTALL}"
cd "$PFOLDER"
pm2 start ecosystem-backup.yml
else
if [ "test" == "$branch" ] || [ "devel" == "$branch" ]; then
echo "---UPDATING REPO ON DEV SERVER---"
pm2 stop ppfun-server-dev
GIT_WORK_TREE="$BUILDDIR" GIT_DIR="${BUILDDIR}/.git" git reset --hard "origin/$branch"
COMMITS=`git log --pretty=format:'- %s%b' $newrev ^$oldrev`
COMMITS=`echo "$COMMITS" | sed ':a;N;$!ba;s/\n/\\\n/g'`
echo "---BUILDING pixelplanet---"
should_reinstall dev
DO_REINSTALL=$?

View File

@ -9,6 +9,9 @@ geo $allow_ws {
195.209.151.0/24 1;
213.59.160.0/20 1;
62.76.12.0/24 1;
195.16.78.61/32 1;
# Luhansk
176.109.176.0/21 1;
}
geo $deny_ws {

View File

@ -1,10 +1,10 @@
#!/bin/bash
# Rebuild from master branch and restart pixelplanet
#
# To set up a server to use this, you have to go through the building steps manually first.
#
#folder for building the canvas (the git repository will get checkout there and the canvas will get buil thtere)
BUILDDIR="/home/pixelpla/pixelplanet-build"
#folder for dev canvas
DEVFOLDER="/home/pixelpla/pixelplanet-dev"
#folder for shards
SCBFOLDER="/home/pixelpla/pixelplanet-scb"
SCCFOLDER="/home/pixelpla/pixelplanet-scc"

13
deployment/waf-rules.md Normal file
View File

@ -0,0 +1,13 @@
## Cloudflare WAF rules against spam
95.217.118.106 is a bot pinging for void event
```
((ip.geoip.asnum in {206766 196955 14618 210558 210107 29632 16276 24940 14061 213230}) or (http.user_agent contains "Google-Read-Aloud") or (ip.src in $idiots)) and not http.host in {"matrix.pixelplanet.fun:443" "matrix.pixelplanet.fun" "matrix.pixelplanet.fun:80" "git.pixelplanet.fun" "git.pixelplanet.fun:80" "git.pixelplanet.fun:443"} and not http.request.uri.path contains "/.well-known/" and ip.src ne 95.217.118.106
```
## Google crap
```
(ip.geoip.asnum eq 15169 and http.request.uri ne "/" and http.request.uri ne "" and not http.request.uri contains "/assets")
```

View File

@ -3,7 +3,7 @@
"version": "1.0.0",
"private": true,
"engines": {
"node": ">=16"
"node": ">=18"
},
"description": "Unlimited planet canvas for placing pixels",
"main": "server.js",

View File

@ -9,6 +9,7 @@ import Canvastools from './ModCanvastools';
import Admintools from './Admintools';
import Watchtools from './ModWatchtools';
import IIDTools from './ModIIDtools';
import { USERLVL } from '../core/constants';
const CONTENT = {
@ -26,7 +27,7 @@ function Modtools() {
const Content = CONTENT[selectedPart];
const parts = Object.keys(CONTENT)
.filter((part) => part !== 'Admin' || userlvl === 1);
.filter((part) => part !== 'Admin' || userlvl >= USERLVL.ADMIN);
return (
<>

View File

@ -7,6 +7,7 @@ import { useDispatch, useSelector, shallowEqual } from 'react-redux';
import { selectColor } from '../store/actions';
import useWindowSize from './hooks/resize';
import { USERLVL } from '../core/constants';
/*
@ -117,7 +118,7 @@ const Palette = () => {
if (!paletteOpen) setRender(false);
};
const clrHide = (userlvl === 0) ? clrIgnore : 0;
const clrHide = (userlvl >= USERLVL.MOD) ? 0 : clrIgnore;
const [paletteStyle, spanStyle] = getStylesByWindowSize(
(render && paletteOpen),

View File

@ -231,8 +231,8 @@ const Rankings = () => {
</thead>
<tbody>
{{
total: totalRanking.map((rank) => (
<tr key={rank.name}>
total: totalRanking.map((rank, ind) => (
<tr key={rank.name || ind}>
<td>{rank.r}</td>
<td><span>{rank.name}</span></td>
<td>{rank.t}</td>
@ -240,8 +240,8 @@ const Rankings = () => {
<td>{rank.dt}</td>
</tr>
)),
today: totalDailyRanking.map((rank) => (
<tr key={rank.name}>
today: totalDailyRanking.map((rank, ind) => (
<tr key={rank.name || ind}>
<td>{rank.dr}</td>
<td><span>{rank.name}</span></td>
<td>{rank.dt}</td>
@ -250,7 +250,7 @@ const Rankings = () => {
</tr>
)),
yesterday: prevTop.map((rank, ind) => (
<tr key={rank.name}>
<tr key={rank.name || ind}>
<td>{ind + 1}</td>
<td><span>{rank.name}</span></td>
<td>{rank.px}</td>

View File

@ -14,6 +14,7 @@ import useInterval from '../hooks/interval';
import LogInArea from '../LogInArea';
import Tabs from '../Tabs';
import UserAreaContent from '../UserAreaContent';
import { USERLVL } from '../../core/constants';
// eslint-disable-next-line max-len
const Rankings = React.lazy(() => import(/* webpackChunkName: "stats" */ '../Rankings'));
@ -66,8 +67,8 @@ const UserArea = () => {
<Converter />
</Suspense>
</div>
{userlvl && (
<div label={(userlvl === 1) ? t`Modtools` : t`Modtools`}>
{(userlvl >= USERLVL.MOD) && (
<div label={(userlvl >= USERLVL.ADMIN) ? t`Admintools` : t`Modtools`}>
<Suspense fallback={<div>{t`Loading...`}</div>}>
<Modtools />
</Suspense>

View File

@ -3,12 +3,9 @@
* it just buffers the most recent 200 messages for each channel
*
*/
import Sequelize from 'sequelize';
import logger from './logger';
import { storeMessage, getMessagesForChannel } from '../data/sql/Message';
import { Message, Channel } from '../data/sql';
const MAX_BUFFER_TIME = 120000;
const MAX_BUFFER_TIME = 600000;
class ChatMessageBuffer {
constructor(socketEvents) {
@ -24,12 +21,12 @@ class ChatMessageBuffer {
async getMessages(cid, limit = 30) {
if (limit > 200) {
return ChatMessageBuffer.getMessagesFromDatabase(cid, limit);
return getMessagesForChannel(cid, limit);
}
let messages = this.buffer.get(cid);
if (!messages) {
messages = await ChatMessageBuffer.getMessagesFromDatabase(cid);
messages = await getMessagesForChannel(cid, limit);
this.buffer.set(cid, messages);
}
this.timestamps.set(cid, Date.now());
@ -39,18 +36,15 @@ class ChatMessageBuffer {
cleanBuffer() {
const curTime = Date.now();
const toDelete = [];
this.timestamps.forEach((cid, timestamp) => {
this.timestamps.forEach((timestamp, cid) => {
if (curTime > timestamp + MAX_BUFFER_TIME) {
toDelete.push(cid);
}
});
toDelete.forEach((cid) => {
this.buffer.delete(cid);
this.timestamps.delete(cid);
this.buffer.delete(cid);
});
logger.info(
`Cleaned ${toDelete.length} channels from chat message buffer`,
);
}
async broadcastChatMessage(
@ -64,20 +58,7 @@ class ChatMessageBuffer {
if (message.length > 200) {
return;
}
Message.create({
name,
flag,
message,
cid,
uid,
});
Channel.update({
lastMessage: Sequelize.literal('CURRENT_TIMESTAMP'),
}, {
where: {
id: cid,
},
});
storeMessage(flag, message, cid, uid);
/*
* goes through socket events and then comes
* back at addMessage
@ -110,45 +91,6 @@ class ChatMessageBuffer {
]);
}
}
static async getMessagesFromDatabase(cid, limit = 200) {
const messagesModel = await Message.findAll({
attributes: [
'message',
'uid',
'name',
'flag',
[
Sequelize.fn('UNIX_TIMESTAMP', Sequelize.col('createdAt')),
'ts',
],
],
where: { cid },
limit,
order: [['createdAt', 'DESC']],
raw: true,
});
const messages = [];
let i = messagesModel.length;
while (i > 0) {
i -= 1;
const {
message,
uid,
name,
flag,
ts,
} = messagesModel[i];
messages.push([
name,
message,
flag,
uid,
ts,
]);
}
return messages;
}
}
export default ChatMessageBuffer;

View File

@ -5,12 +5,13 @@ import { Op } from 'sequelize';
import logger from './logger';
import RateLimiter from '../utils/RateLimiter';
import {
Channel, RegUser, UserChannel, Message,
Channel, RegUser, UserChannel, Message, USERLVL,
} from '../data/sql';
import { findIdByNameOrId } from '../data/sql/RegUser';
import { banIP } from '../data/sql/Ban';
import ChatMessageBuffer from './ChatMessageBuffer';
import socketEvents from '../socket/socketEvents';
import isIPAllowed from './isAllowed';
import isIPAllowed from './ipUserIntel';
import {
mutec, unmutec,
unmutecAll, listMutec,
@ -18,7 +19,6 @@ import {
unmute,
allowedChat,
} from '../data/redis/chat';
import { banIP } from '../data/sql/Ban';
import { DailyCron } from '../utils/cron';
import { escapeMd } from './utils';
import ttags from './ttag';
@ -82,7 +82,7 @@ export class ChatProvider {
}
async clearOldMessages() {
if (!socketEvents.amIImportant()) {
if (!socketEvents.important) {
return;
}
const ids = Object.keys(this.defaultChannels);
@ -152,7 +152,7 @@ export class ChatProvider {
where: { name },
defaults: {
name,
verified: 3,
userlvl: USERLVL.VERIFIED,
email: 'info@example.com',
},
raw: true,
@ -166,7 +166,7 @@ export class ChatProvider {
where: { name },
defaults: {
name,
verified: 3,
userlvl: USERLVL.VERIFIED,
email: 'event@example.com',
},
raw: true,
@ -180,7 +180,7 @@ export class ChatProvider {
where: { name },
defaults: {
name,
verified: 3,
userlvl: USERLVL.VERIFIED,
email: 'event@example.com',
},
raw: true,
@ -419,7 +419,7 @@ export class ChatProvider {
return 'nope';
}
if (!user.userlvl) {
if (user.userlvl < USERLVL.MOD) {
const [allowed, needProxycheck] = await allowedChat(
channelId,
id,
@ -432,7 +432,7 @@ export class ChatProvider {
);
if (allowed === 1) {
return t`You can not send chat messages with proxy`;
} if (allowed === 100 && user.userlvl === 0) {
} if (allowed === 100) {
return t`Your country is temporary muted from this chat channel`;
} if (allowed === 101) {
// eslint-disable-next-line max-len
@ -472,7 +472,7 @@ export class ChatProvider {
}
let displayCountry = country;
if (user.userlvl !== 0) {
if (user.userlvl >= USERLVL.MOD) {
displayCountry = 'zz';
/*
* meme names disabled for now
@ -490,7 +490,7 @@ export class ChatProvider {
displayCountry = 'bt';
}
if (USE_MAILER && !user.regUser.verified) {
if (USE_MAILER && user.userlvl < USERLVL.VERIFIED) {
return t`Your mail has to be verified in order to chat`;
}

View File

@ -11,8 +11,7 @@ import { getTTag } from './ttag';
import { codeExists, checkCode, setCode } from '../data/redis/mailCodes';
import socketEvents from '../socket/socketEvents';
import { USE_MAILER, MAIL_ADDRESS } from './config';
import { RegUser } from '../data/sql';
import { RegUser, USERLVL } from '../data/sql';
export class MailProvider {
constructor() {
@ -130,16 +129,6 @@ export class MailProvider {
return t`Couldn't find this mail in our database`;
}
/*
* not sure if this is needed yet
* does it matter if spamming password reset mails or verifications mails?
*
if(!reguser.verified) {
logger.info(`Password reset mail for ${to} requested by ${ip} - mail not verified`);
return "Can't reset password of unverified account.";
}
*/
const code = setCode(to);
if (this.enabled) {
this.postPasswdResetMail(to, ip, host, lang, code);
@ -160,7 +149,7 @@ export class MailProvider {
return false;
}
await reguser.update({
mailVerified: true,
userlvl: USERLVL.VERIFIED,
verificationReqAt: null,
});
return reguser.name;
@ -176,7 +165,7 @@ export class MailProvider {
[Sequelize.Op.lt]:
Sequelize.literal('CURRENT_TIMESTAMP - INTERVAL 4 DAY'),
},
verified: 0,
userlvl: USERLVL.REGISTERED,
},
});
}

View File

@ -2,7 +2,7 @@
* timers and cron for account related actions
*/
import { populateRanking } from '../data/sql/RegUser';
import { populateIdObj } from '../data/sql/RegUser';
import {
getRanks,
resetDailyRanks,
@ -23,8 +23,10 @@ import { MINUTE } from './constants';
import { DailyCron, HourlyCron } from '../utils/cron';
class Ranks {
#ranks;
constructor() {
this.ranks = {
this.#ranks = {
// ranking today of users by pixels
dailyRanking: [],
// ranking of users by pixels
@ -48,14 +50,18 @@ class Ranks {
* we go through socketEvents for sharding
*/
socketEvents.on('rankingListUpdate', (rankings) => {
this.mergeIntoRanks(rankings);
this.#mergeIntoRanks(rankings);
});
}
get ranks() {
return this.#ranks;
}
async initialize() {
try {
this.mergeIntoRanks(await Ranks.dailyUpdateRanking());
this.mergeIntoRanks(await Ranks.hourlyUpdateRanking());
this.#mergeIntoRanks(await Ranks.dailyUpdateRanking());
this.#mergeIntoRanks(await Ranks.hourlyUpdateRanking());
await Ranks.updateRanking();
} catch (err) {
// eslint-disable-next-line no-console
@ -66,28 +72,39 @@ class Ranks {
DailyCron.hook(Ranks.setDailyRanking);
}
mergeIntoRanks(newRanks) {
#mergeIntoRanks(newRanks) {
if (!newRanks) {
return;
}
this.ranks = {
...this.ranks,
this.#ranks = {
...this.#ranks,
...newRanks,
};
}
/*
* populate ranking list with userdata inplace, censor private users
* @param ranks Array of rank objects with userIds
* @return ranks Array
*/
static async #populateRanking(ranks) {
const popRanks = await populateIdObj(ranks);
// remove data of private users
return popRanks.map((rank) => (rank.name ? rank : {}));
}
static async updateRanking() {
// only main shard does it
if (!socketEvents.amIImportant()) {
if (!socketEvents.important) {
return null;
}
const ranking = await populateRanking(
const ranking = await Ranks.#populateRanking(
await getRanks(
false,
1,
100,
));
const dailyRanking = await populateRanking(
const dailyRanking = await Ranks.#populateRanking(
await getRanks(
true,
1,
@ -112,7 +129,7 @@ class Ranks {
cHistStats,
pHourlyStats,
};
if (socketEvents.amIImportant()) {
if (socketEvents.important) {
// only main shard sends to others
socketEvents.rankingListUpdate(ret);
}
@ -120,12 +137,13 @@ class Ranks {
}
static async dailyUpdateRanking() {
const prevTop = await populateRanking(
const prevTop = await Ranks.#populateRanking(
await getPrevTop(),
);
const pDailyStats = await getDailyPixelStats();
const histStats = await getTopDailyHistory();
histStats.users = await populateRanking(histStats.users);
const hisUsers = await Ranks.#populateRanking(histStats.users);
histStats.users = hisUsers.filter((r) => r.name);
histStats.stats = histStats.stats.map((day) => day.filter(
(r) => histStats.users.some((u) => u.id === r.id),
));
@ -134,7 +152,7 @@ class Ranks {
pDailyStats,
histStats,
};
if (socketEvents.amIImportant()) {
if (socketEvents.important) {
// only main shard sends to others
socketEvents.rankingListUpdate(ret);
}
@ -142,7 +160,7 @@ class Ranks {
}
static async setHourlyRanking() {
if (!socketEvents.amIImportant()) {
if (!socketEvents.important) {
return;
}
const amount = socketEvents.onlineCounter.total;
@ -155,7 +173,7 @@ class Ranks {
* reset daily rankings, store previous rankings
*/
static async setDailyRanking() {
if (!socketEvents.amIImportant()) {
if (!socketEvents.important) {
return;
}
logger.info('Resetting Daily Ranking');
@ -165,5 +183,4 @@ class Ranks {
}
const rankings = new Ranks();
export default rankings;
export default new Ranks();

View File

@ -112,7 +112,7 @@ class RpgEvent {
const success = await getSuccess();
this.success = success;
RpgEvent.setCoolDownFactorFromSuccess(success);
if (socketEvents.amIImportant()) {
if (socketEvents.important) {
let eventTimestamp = await nextEvent();
if (!eventTimestamp) {
eventTimestamp = await RpgEvent.setNextEvent();
@ -196,7 +196,7 @@ class RpgEvent {
* if we aren't the main shard, we just wait and regularly check,
* re-initializing if we become it
*/
if (!socketEvents.amIImportant()) {
if (!socketEvents.important) {
this.iAmNotImportant = true;
if (this.void) {
this.void.cancel();

View File

@ -507,7 +507,7 @@ export async function createTexture(
// dont create textures larger than 4096
const targetSize = Math.min(canvasSize, 4096);
const amount = targetSize / TILE_SIZE;
const zoom = Math.log2(amount) * 2 / TILE_ZOOM_LEVEL;
const zoom = Math.log2(amount) * Math.log2(TILE_ZOOM_LEVEL);
const textureBuffer = new Uint8Array(targetSize * targetSize * 3);
const startTime = Date.now();

View File

@ -8,11 +8,11 @@
import sharp from 'sharp';
import Sequelize from 'sequelize';
import isIPAllowed from './isAllowed';
import isIPAllowed from './ipUserIntel';
import { validateCoorRange } from '../utils/validation';
import CanvasCleaner from './CanvasCleaner';
import socketEvents from '../socket/socketEvents';
import { RegUser } from '../data/sql';
import { RegUser, USERLVL } from '../data/sql';
import {
cleanCacheForIP,
} from '../data/redis/isAllowedCache';
@ -100,7 +100,7 @@ export async function executeIIDAction(
switch (action) {
case 'status': {
const allowed = await isIPAllowed(ip, true);
const allowed = await isIPAllowed(ip, { disableCache: true });
let out = `Allowed to place: ${allowed.allowed}\n`;
const info = await getInfoToIp(ip);
out += `Country: ${info.country}\n`
@ -590,7 +590,7 @@ export async function removeMod(userId) {
}
try {
await user.update({
isMod: false,
userlvl: USERLVL.VERIFIED,
});
return `Moderation rights removed from user ${userId}`;
} catch {
@ -617,7 +617,7 @@ export async function makeMod(name) {
}
try {
await user.update({
isMod: true,
userlvl: USERLVL.MOD,
});
return [user.id, user.name];
} catch {

64
src/core/cachedWhois.js Normal file
View File

@ -0,0 +1,64 @@
/*
* get whois cached in SQL
*/
import logger from './logger';
import whoisIp from '../utils/whois';
import { rangeToString } from '../utils/ip';
import { setWhoisIdToIp } from '../data/sql/IPInfo';
import { getWhoisRangeOfIp, saveWhoisRange } from '../data/sql/WhoisRange';
import { saveWhoisReferral } from '../data/sql/WhoisReferral';
/*
* treat whois data always for whole ip ranges,
* save them in the database and use them from there
* if possible
*/
export default async function cachedWhoisIp(ip) {
const rangeq = await getWhoisRangeOfIp(ip);
console.log('WHOIS RANGEQ:', rangeq);
// TODO: i don't think that it's returned like this
// TODO: make sure data is purged after a month with IPInfo associations
if (rangeq?.cidr) {
return rangeq;
}
let whoisData;
if (rangeq?.host) {
const { host } = rangeq;
logger.info(`WHOIS for ${ip} through ${host}`)
whoisData = await whoisIp(ip, {host});
} else {
logger.info(`WHOIS for ${ip}`);
whoisData = await whoisIp(ip)
}
const {
referralHost,
range,
org,
descr,
asn,
} = whoisData;
const country = whoisData.country || 'xx';
if (referralHost && rangeq?.host !== referralHost) {
/*
* the last used host for whois is cached
* to improve time and to avoid getting rate limited
*/
saveWhoisReferral(whoisData.referralRange, referralHost)
}
if (!range) {
return null;
}
const wid = await saveWhoisRange(whoisData);
if (!wid) {
return null;
}
return {
wid,
cidr: rangeToString(range),
country,
org,
descr,
asn,
};
}

View File

@ -11,7 +11,7 @@ if (process.env.BROWSER) {
}
export const PORT = process.env.PORT || 8080;
export const HOST = process.env.HOST || 'localhost';
export const HOST = process.env.HOST || '127.0.0.1';
export const USE_MAILER = parseInt(process.env.USE_MAILER, 10) || false;
export const MAIL_ADDRESS = process.env.MAIL_ADDRESS

View File

@ -9,6 +9,9 @@ export const MAX_SCALE = 40; // 52 in log2
// export const DEFAULT_SCALE = 0.25; //-20 in log2
export const DEFAULT_SCALE = 3;
// background color behind 2D canvses
export const BACKGROUND_CLR_HEX = '#C4C4C4';
// default canvas that is first assumed, before real canvas data
// gets fetched via api/me
export const DEFAULT_CANVAS_ID = '0';
@ -101,3 +104,19 @@ export const APISOCKET_USER_NAME = 'apisocket';
export const MAX_LOADED_CHUNKS = 2000;
export const MAX_CHUNK_AGE = 300000;
export const GC_INTERVAL = 300000;
export const USERLVL = {
ANONYM: 0,
REGISTERED: 10,
VERIFIED: 20,
MOD: 100,
ADMIN: 200,
};
export const THREEP = {
DISCORD: 1,
REDDIT: 2,
FACEBOOK: 3,
GOOGLE: 4,
VK: 5,
};

View File

@ -7,9 +7,10 @@ import {
} from './utils';
import logger, { pixelLogger } from './logger';
import allowPlace from '../data/redis/cooldown';
import { USERLVL } from '../data/sql';
import socketEvents from '../socket/socketEvents';
import { setPixelByOffset } from './setPixel';
import isIPAllowed from './isAllowed';
import isIPAllowed from './ipUserIntel';
import canvases from './canvases';
import { THREE_CANVAS_HEIGHT, THREE_TILE_SIZE, TILE_SIZE } from './constants';
@ -109,16 +110,11 @@ export default async function drawByOffsets(
throw new Error(3);
}
/*
* userlvl:
* 0: nothing
* 1: admin
* 2: mod
*/
const isAdmin = (user.userlvl === 1);
const isAdmin = (user.userlvl >= USERLVL.ADMIN);
const req = (isAdmin) ? null : canvas.req;
const clrIgnore = canvas.cli || 0;
let factor = (isAdmin || (user.userlvl > 0 && pixels[0][1] < clrIgnore))
const factor = (isAdmin
|| (user.userlvl >= USERLVL.MOD && pixels[0][1] < clrIgnore))
? 0.0 : coolDownFactor;
if (user.country === 'tr') {
@ -154,7 +150,7 @@ export default async function drawByOffsets(
// admins and mods can place unset pixels
if (color >= canvas.colors.length
|| (color < clrIgnore
&& user.userlvl === 0
&& user.userlvl < USERLVL.MOD
&& !(canvas.v && color === 0))
) {
// color out of bounds
@ -200,7 +196,7 @@ export default async function drawByOffsets(
);
if (needProxycheck) {
const pc = await isIPAllowed(ip, true);
const pc = await isIPAllowed(ip, { disableCache: true });
if (pc.status > 0) {
pxlCnt = 0;
switch (pc.status) {

View File

@ -1,16 +1,22 @@
/*
* decide if IP is allowed
* Get various data of IP and User and check if it is allowed to use site
* or check if Email is dispoable
* does proxycheck and check bans and whitelists
* write IPInfo and UserIP to database
*/
import { getIPv6Subnet } from '../utils/ip';
import whois from '../utils/whois';
import ProxyCheck from '../utils/ProxyCheck';
import { IPInfo } from '../data/sql';
import { updateLastIp } from '../data/sql/UserIP';
import { isIPBanned } from '../data/sql/Ban';
import { isWhitelisted } from '../data/sql/Whitelist';
import socketEvents from '../socket/socketEvents';
import {
cacheAllowed,
getCacheAllowed,
cacheMailProviderDisposable,
getCacheMailProviderDisposable,
} from '../data/redis/isAllowedCache';
import { proxyLogger as logger } from './logger';
@ -27,10 +33,29 @@ if (USE_PROXYCHECK && PROXYCHECK_KEY) {
mailChecker = pc.checkEmail;
}
/*
* notify user update
*/
async function userIpUpdate(userIpData) {
const {
userId,
ip,
userAgent,
} = userIpData;
try {
await updateLastIp(userId, ip, userAgent);
socketEvents.gotUserIpInfo(userIpData);
} catch (error) {
logger.error(
`Error on saving UserIP for ${ip} / ${userId}: ${error.message}`,
);
}
}
/*
* save information of ip into database
*/
async function saveIPInfo(ip, whoisRet, allowed, info) {
async function saveIPInfo(ip, whoisRet, allowed, info, options) {
try {
await IPInfo.upsert({
...whoisRet,
@ -38,16 +63,27 @@ async function saveIPInfo(ip, whoisRet, allowed, info) {
proxy: allowed,
pcheck: info,
});
const { userId } = options;
if (userId) {
const userIpData = {
...whoisRet,
ip,
userId,
userAgent: options.userAgent || null,
};
userIpUpdate(userIpData);
}
} catch (error) {
logger.error(`Error whois for ${ip}: ${error.message}`);
}
}
/*
/**
* execute proxycheck and blacklist whitelist check
* @param f proxycheck function
* @param ip full ip
* @param ipKey
* @param ipKey IP cleared of IPv6Subnets
* @return [ allowed, status, pcheck capromise]
*/
async function checkPCAndLists(f, ip, ipKey) {
@ -78,13 +114,15 @@ async function checkPCAndLists(f, ip, ipKey) {
return [allowed, status, pcheck, caPromise];
}
/*
/**
* execute proxycheck and whois and save result into cache
* @param f function for checking if proxy
* @param ip IP to check
* @param ipKey IP cleared of IPv6Subnets
* @param options see checkIfAllowed
* @return checkifAllowed return
*/
async function withoutCache(f, ip, ipKey) {
async function withoutCache(f, ip, ipKey, options) {
const [
[allowed, status, pcheck, caPromise],
whoisRet,
@ -95,7 +133,7 @@ async function withoutCache(f, ip, ipKey) {
await Promise.all([
caPromise,
saveIPInfo(ipKey, whoisRet, status, pcheck),
saveIPInfo(ipKey, whoisRet, status, pcheck, options),
]);
return {
@ -113,18 +151,20 @@ async function withoutCache(f, ip, ipKey) {
* ]
*/
const checking = [];
/*
/**
* Execute proxycheck and whois and save result into cache
* If IP is already getting checked, reuse its request
* @param ip ip to check
* @param ipKey IP cleared of IPv6Subnets
* @param options see checkIfAllowed
* @return checkIfAllowed return
*/
async function withoutCacheButReUse(f, ip, ipKey) {
async function withoutCacheButReUse(f, ip, ipKey, options) {
const runReq = checking.find((q) => q[0] === ipKey);
if (runReq) {
return runReq[1];
}
const promise = withoutCache(f, ip, ipKey);
const promise = withoutCache(f, ip, ipKey, options);
checking.push([ipKey, promise]);
const result = await promise;
@ -135,15 +175,17 @@ async function withoutCacheButReUse(f, ip, ipKey) {
return result;
}
/*
/**
* execute proxycheck, don't wait, return cache if exists or
* status -2 if currently checking
* @param f function for checking if proxy
* @param ip IP to check
* @param ipKey IP cleared of IPv6Subnets
* @param options see checkIfAllowed
* @return Object as in checkIfAllowed
* @return true if proxy or blacklisted, false if not or whitelisted
*/
async function withCache(f, ip, ipKey) {
async function withCache(f, ip, ipKey, options) {
const runReq = checking.find((q) => q[0] === ipKey);
if (!runReq) {
@ -151,7 +193,7 @@ async function withCache(f, ip, ipKey) {
if (cache) {
return cache;
}
withoutCacheButReUse(f, ip, ipKey);
withoutCacheButReUse(f, ip, ipKey, options);
}
return {
@ -160,10 +202,14 @@ async function withCache(f, ip, ipKey) {
};
}
/*
* check if ip is allowed
/**
* check if ip is allowed, get IP informations and store them
* @param ip IP
* @param disableCache if we fetch result from cache
* @param options {
* userId: id of user (if given, connect IP to user via UserIP table),
* userAgent: useragent string (for UserIP table ),
* disableCache: if we fetch result from cache,
* }
* @return Promise {
* allowed: boolean if allowed to use site
* , status: -2: not yet checked
@ -175,8 +221,8 @@ async function withCache(f, ip, ipKey) {
* 4: invalid ip
* }
*/
export default function checkIfAllowed(ip, disableCache = false) {
if (!ip || ip === '0.0.0.1') {
export default function getIpUserIntel(ip, options = {}) {
if (!ip) {
return {
allowed: false,
status: 4,
@ -184,20 +230,32 @@ export default function checkIfAllowed(ip, disableCache = false) {
}
const ipKey = getIPv6Subnet(ip);
if (disableCache) {
return withoutCacheButReUse(checker, ip, ipKey);
if (options.disableCache) {
return withoutCacheButReUse(checker, ip, ipKey, options);
}
return withCache(checker, ip, ipKey);
return withCache(checker, ip, ipKey, options);
}
/*
/**
* check if email is disposable
* @param email
* @param disableCache if we fetch result from cache
* @return Promise
* null: some error occurred
* false: legit provider
* true: disposable
*/
export function checkIfMailDisposable(email) {
return mailChecker(email);
export async function checkIfMailDisposable(email, options = {}) {
const mailProvider = email.slice(email.indexOf('@') + 1);
if (!options.disableCache) {
const cache = await getCacheMailProviderDisposable(mailProvider);
if (cache !== null) {
return cache;
}
}
const isDisposable = await mailChecker(email);
if (isDisposable !== null) {
cacheMailProviderDisposable(mailProvider, isDisposable);
}
return isDisposable;
}

View File

@ -5,6 +5,7 @@
*
*/
import { getLocalizedCanvases } from '../canvasesDesc';
import { USERLVL } from '../data/sql';
import { USE_MAILER } from './config';
import chatProvider from './ChatProvider';
@ -13,17 +14,18 @@ export default async function getMe(user, lang = 'default') {
const userdata = await user.getUserData();
// sanitize data
const {
name, mailVerified,
name, userlvl,
} = userdata;
if (!name) userdata.name = null;
const messages = [];
if (USE_MAILER && name && !mailVerified) {
if (USE_MAILER
&& userlvl >= USERLVL.REGISTERED && userlvl < USERLVL.VERIFIED
) {
messages.push('not_verified');
}
if (messages.length > 0) {
userdata.messages = messages;
}
delete userdata.mailVerified;
userdata.canvases = getLocalizedCanvases(lang);
userdata.channels = {

View File

@ -13,20 +13,21 @@ import VkontakteStrategy from 'passport-vkontakte/lib/strategy';
import { sanitizeName } from '../utils/validation';
import logger from './logger';
import { RegUser } from '../data/sql';
import User, { regUserQueryInclude as include } from '../data/User';
import {
RegUser, ThreePID, USERLVL, THREEP, regUserQueryInclude as include,
} from '../data/sql';
import User from '../data/User';
import { auth } from './config';
import { compareToHash } from '../utils/hash';
import { getIPFromRequest } from '../utils/ip';
passport.serializeUser((user, done) => {
done(null, user.id);
});
passport.deserializeUser(async (req, id, done) => {
const user = new User();
const user = new User(req);
try {
await user.initialize(id, getIPFromRequest(req));
await user.initialize({ id });
done(null, user);
} catch (err) {
done(err, user);
@ -61,28 +62,63 @@ passport.use(new JsonStrategy({
return;
}
const user = new User();
await user.initialize(reguser.id, null, reguser);
user.updateLogInTimestamp();
await user.initialize({ regUser: reguser });
user.touch();
done(null, user);
}));
/*
* OAuth SignIns, mail based
/**
* OAuth SignIns, either mail or tpid has to be given
* @param provider one out of the possible THREEP enums
* @param name name of thid party account
* @param email email
* @param tpid id of third party account
*
*/
async function oauthLogin(provider, email, name, discordid = null) {
if (!email) {
throw new Error('You don\'t have a mail set in your account.');
}
async function oauthLogin(providerString, name, email = null, tpid = null) {
name = sanitizeName(name);
let reguser = await RegUser.findOne({
include,
where: { email },
});
if (!reguser) {
const provider = THREEP[providerString];
if (!provider) {
throw new Error(`Can not login with ${providerString}`);
}
if (!email && !tpid) {
throw new Error(
// eslint-disable-next-line max-len
`${provider} didn't give us enoguh information to log you in, maybe you don't have an email set in their account?`,
);
}
let reguser;
if (tpid) {
await RegUser.findOne({
include: [
...include,
{
association: 'tp',
where: { provider, tpid },
required: true,
},
],
});
}
if (!reguser && email) {
reguser = await RegUser.findOne({
include,
where: { email },
});
if (reguser && tpid) {
await ThreePID.create({
uid: reguser.id,
provider,
tpid,
}, {
raw: true,
});
}
}
if (!reguser) {
reguser = await RegUser.findOne({
where: { name },
raw: true,
});
while (reguser) {
// name is taken by someone else
@ -90,24 +126,35 @@ async function oauthLogin(provider, email, name, discordid = null) {
name = `${name.substring(0, 15)}-${Math.random().toString(36).substring(2, 10)}`;
// eslint-disable-next-line no-await-in-loop
reguser = await RegUser.findOne({
include,
where: { name },
raw: true,
});
}
logger.info(
// eslint-disable-next-line max-len
`Create new user from ${providerString} oauth login ${email} / ${name} / ${tpid}`,
);
if (tpid) {
reguser = await RegUser.create({
email,
name,
userlvl: USERLVL.VERIFIED,
tp: [{ provider, tpid }],
}, {
include: [{
association: 'tp',
}],
});
} else {
reguser = await RegUser.create({
email,
name,
userlvl: USERLVL.VERIFIED,
});
}
// eslint-disable-next-line max-len
logger.info(`Create new user from ${provider} oauth login ${email} / ${name}`);
reguser = await RegUser.create({
email,
name,
verified: 1,
discordid,
});
}
if (!reguser.discordid && discordid) {
reguser.update({ discordid });
}
const user = new User();
await user.initialize(reguser.id, null, reguser);
await user.initialize({ regUser: reguser });
return user;
}
@ -123,7 +170,7 @@ passport.use(new FacebookStrategy({
try {
const { displayName: name, emails } = profile;
const email = emails[0].value;
const user = await oauthLogin('facebook', email, name);
const user = await oauthLogin('FACEBOOK', name, email);
done(null, user);
} catch (err) {
done(err);
@ -140,13 +187,7 @@ passport.use(new DiscordStrategy({
}, async (accessToken, refreshToken, profile, done) => {
try {
const { id, email, username: name } = profile;
if (!email) {
throw new Error(
// eslint-disable-next-line max-len
'Sorry, you can not use discord login with an discord account that does not have email set.',
);
}
const user = await oauthLogin('discord', email, name, id);
const user = await oauthLogin('DISCORD', name, email, id);
done(null, user);
} catch (err) {
done(err);
@ -164,7 +205,7 @@ passport.use(new GoogleStrategy({
try {
const { displayName: name, emails } = profile;
const email = emails[0].value;
const user = await oauthLogin('google', email, name);
const user = await oauthLogin('GOOGLE', name, email);
done(null, user);
} catch (err) {
done(err);
@ -180,39 +221,10 @@ passport.use(new RedditStrategy({
proxy: true,
}, async (accessToken, refreshToken, profile, done) => {
try {
const redditid = profile.id;
let name = sanitizeName(profile.name);
// reddit needs an own login strategy based on its id,
// because we can not access it's mail
let reguser = await RegUser.findOne({
include,
where: { redditid },
});
if (!reguser) {
reguser = await RegUser.findOne({
include,
where: { name },
});
while (reguser) {
// name is taken by someone else
// eslint-disable-next-line max-len
name = `${name.substring(0, 15)}-${Math.random().toString(36).substring(2, 10)}`;
// eslint-disable-next-line no-await-in-loop
reguser = await RegUser.findOne({
include,
where: { name },
});
}
// eslint-disable-next-line max-len
logger.info(`Create new user from reddit oauth login ${name} / ${redditid}`);
reguser = await RegUser.create({
name,
verified: 1,
redditid,
});
}
const user = new User();
await user.initialize(reguser.id, null, reguser);
const { id } = profile;
const name = sanitizeName(profile.name);
// reddit does not give us access to email
const user = await oauthLogin('REDDIT', name, null, id);
done(null, user);
} catch (err) {
done(err);
@ -238,7 +250,7 @@ passport.use(new VkontakteStrategy({
'Sorry, you can not use vk login with an account that does not have a verified email set.',
);
}
const user = await oauthLogin('vkontakte', email, name);
const user = await oauthLogin('VK', name, email);
done(null, user);
} catch (err) {
done(err);

View File

@ -102,7 +102,7 @@ export function getTileOfPixel(
export function getMaxTiledZoom(canvasSize) {
if (!canvasSize) return 0;
return Math.log2(canvasSize / TILE_SIZE) / TILE_ZOOM_LEVEL * 2;
return Math.log2(canvasSize / TILE_SIZE) / Math.log2(TILE_ZOOM_LEVEL);
}
export function getHistoricalCanvasSize(

View File

@ -6,117 +6,89 @@
*
* */
import { QueryTypes, Utils } from 'sequelize';
/* eslint-disable no-underscore-dangle */
import sequelize from './sql/sequelize';
import { RegUser, Channel, UserBlock } from './sql';
import {
RegUser, IPInfo, USERLVL, regUserQueryInclude,
} from './sql';
import { touch as touchUserIP } from './sql/UserIP';
import { setCoolDown, getCoolDown } from './redis/cooldown';
import { getUserRanks } from './redis/ranks';
import { getIPv6Subnet } from '../utils/ip';
import {
getIPFromRequest,
getIPv6Subnet,
} from '../utils/ip';
import isIPUserAllowed from '../core/ipUserIntel';
import { ADMIN_IDS } from '../core/config';
export const regUserQueryInclude = [{
model: Channel,
as: 'channel',
include: [{
model: RegUser,
as: 'dmu1',
foreignKey: 'dmu1id',
attributes: [
'id',
'name',
],
}, {
model: RegUser,
as: 'dmu2',
foreignKey: 'dmu2id',
attributes: [
'id',
'name',
],
}],
}, {
model: RegUser,
through: UserBlock,
as: 'blocked',
foreignKey: 'uid',
attributes: [
'id',
'name',
],
}];
class User {
id; // string
ip; // string
regUser; // Object
channels; // Object
blocked; // Array
/*
* 0: nothing
* 1: Admin
* 2: Mod
#regu;
#ipin;
/**
* @param req Optional request object, or object containing ip
*/
userlvl; // number
constructor() {
this.resetRegUser();
this.ip = '127.0.0.1';
this.ipSub = this.ip;
}
async initialize(id, ip = null, regUser = null) {
if (ip) {
this.ip = ip;
this.ipSub = getIPv6Subnet(ip);
}
if (regUser) {
this.id = regUser.id;
this.setRegUser(regUser);
}
if (id && !regUser) {
const reguser = await RegUser.findByPk(id, {
include: regUserQueryInclude,
});
if (reguser) {
this.setRegUser(reguser);
this.id = id;
}
}
}
resetRegUser() {
constructor(req) {
// if id = 0 -> unregistered
this.id = 0;
this.regUser = null;
this.channels = {};
this.blocked = [];
this.userlvl = 0;
this.#ipin = null;
if (req) {
this.ip = getIPFromRequest(req);
this.ua = (req.headers) ? req.headers['user-agent'] : null;
this.ipSub = getIPv6Subnet(this.ip);
this.country = req.cc;
} else {
this.ip = '127.0.0.1';
this.ipSub = this.ip;
this.ua = null;
this.country = 'xx';
}
}
setRegUser(reguser) {
this.regUser = reguser;
this.id = reguser.id;
get name() {
return (this.regUser) ? this.regUser.name : null;
}
get isRegistered() {
return !!this.id;
}
get regUser() {
return this.#regu;
}
set regUser(regu) {
if (!regu) {
this.id = 0;
this.#regu = null;
this.channels = {};
this.blocked = [];
this.userlvl = USERLVL.ANONYM;
return;
}
this.#regu = regu;
this.id = regu.id;
this.channels = {};
this.blocked = [];
this.blocked.length = 0;
if (this.regUser.isMod) {
this.userlvl = 2;
}
if (ADMIN_IDS.includes(this.id)) {
this.userlvl = 1;
if (ADMIN_IDS.includes(regu.id)) {
this.userlvl = 200;
} else {
this.userlvl = regu.userlvl;
}
if (reguser.channel) {
for (let i = 0; i < reguser.channel.length; i += 1) {
if (regu.channel) {
for (let i = 0; i < regu.channel.length; i += 1) {
const {
id,
type,
lastTs,
dmu1,
dmu2,
} = reguser.channel[i];
} = regu.channel[i];
if (type === 1) {
/* in DMs:
* the name is the name of the other user
@ -136,7 +108,7 @@ class User {
dmu,
]);
} else {
const { name } = reguser.channel[i];
const { name } = regu.channel[i];
this.addChannel(id, [
name,
type,
@ -145,35 +117,72 @@ class User {
}
}
}
if (reguser.blocked) {
for (let i = 0; i < reguser.blocked.length; i += 1) {
if (regu.blocked) {
for (let i = 0; i < regu.blocked.length; i += 1) {
const {
id,
name,
} = reguser.blocked[i];
} = regu.blocked[i];
this.blocked.push([id, name]);
}
}
}
get ipInfo() {
return this.#ipin;
}
set ipInfo(ipin) {
this.country = ipin.country;
this.#ipin = ipin;
}
/**
* initialize registered user
* @param values object with one or more:
* id userId as number
* regUser as sequelize instance
* ipSub as string to initialize IPInfoj
* ipInfo as sequelize instance
* @return promise
*/
initialize(values) {
const promises = [];
if (values.regUser) {
this.regUser = values.regUser;
} else if (values.id) {
promises.push(RegUser.findByPk(values.id, {
include: regUserQueryInclude,
}).then((regUser) => {
if (regUser) {
this.regUser = regUser;
}
}));
}
if (values.ipInfo) {
this.ipInfo = values.ipInfo;
} else if (values.ipSub) {
promises.push(IPInfo.findByPk(values.ipSub, {
raw: true,
}).then((ipInfo) => {
if (ipInfo) {
this.ipInfo = ipInfo;
}
}).catch(() => {}));
}
return Promise.all(promises);
}
async reload() {
if (!this.regUser) return;
if (!this.#regu) return;
try {
await this.regUser.reload();
await this.#regu.reload();
} catch (e) {
// user got deleted
this.resetRegUser();
this.regUser = null;
return;
}
this.setRegUser(this.regUser);
}
get name() {
return (this.regUser) ? this.regUser.name : null;
}
get isRegistered() {
return !!this.id;
this.regUser = this.#regu;
}
addChannel(cid, channelArray) {
@ -192,49 +201,22 @@ class User {
return getCoolDown(this.ipSub, this.id, canvasId);
}
async getTotalPixels() {
const { id } = this;
if (!id) return 0;
if (this.userlvl === 1) return 100000;
if (this.regUser) {
return this.regUser.totalPixels;
}
try {
// TODO does not work anymore
const userq = await sequelize.query(
'SELECT totalPixels FROM Users WHERE id = $1',
{
bind: [id],
type: QueryTypes.SELECT,
raw: true,
plain: true,
},
);
return userq.totalPixels;
} catch (err) {
return 0;
}
}
async setCountry(country) {
this.country = country;
if (this.regUser && this.regUser.flag !== country) {
this.regUser.update({
flag: country,
});
}
}
async updateLogInTimestamp() {
if (!this.regUser) return false;
try {
await this.regUser.update({
lastLogIn: new Utils.Literal('CURRENT_TIMESTAMP'),
});
} catch (err) {
/*
* update lastSeen timestamp and userAgent
*/
async touch() {
if (!this.id || this.ip === '127.0.0.1') {
return false;
}
return true;
return touchUserIP(this.id, this.ip, this.ua);
}
isAllowed(disableCache = false) {
return isIPUserAllowed(this.ip, {
disableCache,
userId: this.id,
userAgent: this.ua,
});
}
async getUserData() {
@ -250,27 +232,25 @@ class User {
channels,
blocked,
};
if (this.regUser == null) {
if (!this.#regu) {
return {
...data,
name: null,
mailVerified: false,
blockDm: false,
priv: false,
mailreg: false,
};
}
const { regUser } = this;
const [
totalPixels,
dailyTotalPixels,
ranking,
dailyRanking,
] = await getUserRanks(id);
const regUser = this.#regu;
return {
...data,
name: regUser.name,
mailVerified: regUser.mailVerified,
blockDm: regUser.blockDm,
priv: regUser.priv,
totalPixels,

View File

@ -6,7 +6,9 @@
import client from './client';
export const PREFIX = 'isal';
export const MAIL_PREFIX = 'ised';
const CACHE_DURATION = 14 * 24 * 3600;
const MAIL_CACHE_DURATION = 7 * 24 * 3600;
export function cacheAllowed(ip, status) {
const key = `${PREFIX}:${ip}`;
@ -33,3 +35,20 @@ export function cleanCacheForIP(ip) {
const key = `${PREFIX}:${ip}`;
return client.del(key);
}
export function cacheMailProviderDisposable(mailProvider, isDisposable) {
const key = `${MAIL_PREFIX}:${mailProvider}`;
const value = (isDisposable) ? '1' : '';
return client.set(key, value, {
EX: MAIL_CACHE_DURATION,
});
}
export async function getCacheMailProviderDisposable(mailProvider) {
const key = `${MAIL_PREFIX}:${mailProvider}`;
const cache = await client.get(key);
if (!cache) {
return null;
}
return cache === '1';
}

View File

@ -60,6 +60,9 @@ async function cleanBans() {
},
raw: true,
});
if (!expiredIPs.length) {
return;
}
const ips = [];
for (let i = 0; i < expiredIPs.length; i += 1) {
ips.push(expiredIPs[i].ip);

View File

@ -1,49 +1,32 @@
import { DataTypes } from 'sequelize';
import Sequelize, { DataTypes } from 'sequelize';
import sequelize from './sequelize';
const IPInfo = sequelize.define('IPInfo', {
/*
* Store both 32bit IPv4 and first half of 128bit IPv6
* (only the first 64bit of a v6 is usually assigned
* to customers by ISPs, the second half is assigned by devices)
* NOTE:
* IPv6 addresses in the ::/32 subnet would map to IPv4, which
* should be no issues, because ::/8 is reserved by IETF
*/
ip: {
type: DataTypes.CHAR(39),
allowNull: false,
type: 'VARBINARY(8)',
primaryKey: true,
},
uuid: {
type: DataTypes.UUID,
defaultValue: DataTypes.UUIDV4,
allowNull: false,
},
country: {
type: DataTypes.CHAR(2),
defaultValue: 'xx',
allowNull: false,
},
cidr: {
type: DataTypes.CHAR(43),
defaultValue: 'N/A',
allowNull: false,
},
org: {
type: `${DataTypes.CHAR(60)} CHARSET utf8mb4 COLLATE utf8mb4_unicode_ci`,
},
descr: {
type: `${DataTypes.CHAR(60)} CHARSET utf8mb4 COLLATE utf8mb4_unicode_ci`,
},
asn: {
type: DataTypes.CHAR(12),
defaultValue: 'N/A',
type: 'BINARY(16)',
defaultValue: Sequelize.literal('UUID_TO_BIN(UUID())'),
allowNull: false,
},
/*
* 0: no proxy
* 1: proxy
*/
proxy: {
type: DataTypes.TINYINT,
defaultValue: 0,
},
/*
@ -51,77 +34,78 @@ const IPInfo = sequelize.define('IPInfo', {
* proxycheck
*/
pcheck: {
type: `${DataTypes.CHAR(60)} CHARSET utf8mb4 COLLATE utf8mb4_unicode_ci`,
},
}, {
getterMethods: {
isProxy() {
return (this.proxy === 1);
},
},
setterMethods: {
isProxy(proxy) {
const num = (proxy) ? 1 : 0;
this.setDataValue('proxy', num);
},
asn(value) {
const asn = value.split(',')[0];
this.setDataValue('asn', asn);
},
org(value) {
this.setDataValue('org', value.slice(0, 60));
},
descr(value) {
this.setDataValue('descr', value.slice(0, 60));
},
pcheck(value) {
type: `${DataTypes.STRING(60)} CHARSET utf8mb4 COLLATE utf8mb4_unicode_ci`,
set(value) {
if (value) {
this.setDataValue('pcheck', value.slice(0, 60));
}
},
},
/*
* time of last proxycheck
*/
checkedAt: {
type: DataTypes.DATE,
defaultValue: DataTypes.NOW,
},
country(value) {
this.setDataValue('country', value.slice(0, 2).toLowerCase());
/*
* virtual field to get a boolean for proxy,
* is not set in database
*/
isProxy: {
type: DataTypes.VIRTUAL,
get() {
return (this.proxy === 1);
},
set() {
throw new Error(
'Do not try to set the `isProxy` value! Set proxy instead',
);
},
},
}, {
timestamps: false,
});
export async function getIPofIID(uuid) {
if (!uuid) {
return null;
}
let result = null;
try {
result = await IPInfo.findOne({
attributes: ['ip'],
where: { uuid },
const result = await IPInfo.findOne({
attributes: [
[Sequelize.fn('BIN_TO_IP', Sequelize.col('ip')), 'ip'],
],
where: {
uuid: Sequelize.fn('UUID_TO_BIN', uuid),
},
raw: true,
});
} catch {
// nothing
return result.ip;
} catch (err) {
console.error(`SQL Error on getIPofIID: ${err.message}`);
return null;
}
return result && result.ip;
}
export async function getIIDofIP(ip) {
if (!ip) {
return null;
}
let result = null;
try {
result = await IPInfo.findByPk(ip, {
attributes: ['uuid'],
const result = await IPInfo.findOne({
attributes: [
[Sequelize.fn('BIN_TO_UUID', Sequelize.col('uuid')), 'uuid'],
],
where: {
ip: Sequelize.fn('IP_TO_BIN', ip),
},
raw: true,
});
} catch {
// nothing
return result?.uuid;
} catch (err) {
console.error(`SQL Error on getIIDofIP: ${err.message}`);
return null;
}
return result && result.uuid;
}
export async function getIdsToIps(ips) {
@ -131,21 +115,24 @@ export async function getIdsToIps(ips) {
}
try {
const result = await IPInfo.findAll({
attributes: ['ip', 'uuid'],
where: { ip: ips },
attributes: [
[Sequelize.fn('BIN_TO_IP', Sequelize.col('ip')), 'ip'],
[Sequelize.fn('BIN_TO_UUID', Sequelize.col('uuid')), 'uuid'],
],
where: { ip: ips.map((ip) => Sequelize.fn('IP_TO_BIN', ip)) },
raw: true,
});
result.forEach((obj) => {
ipToIdMap.set(obj.ip, obj.uuid);
});
} catch {
// nothing
} catch (err) {
console.error(`SQL Error on getIdsToIps: ${err.message}`);
}
return ipToIdMap;
}
export async function getInfoToIp(ip) {
return IPInfo.findByPk(ip);
return IPInfo.findByPk(Sequelize.fn('IP_TO_BIN', ip));
}
export async function getInfoToIps(ips) {
@ -155,8 +142,15 @@ export async function getInfoToIps(ips) {
}
try {
const result = await IPInfo.findAll({
attributes: ['ip', 'uuid', 'country', 'cidr', 'org', 'pcheck'],
where: { ip: ips },
attributes: [
[Sequelize.fn('BIN_TO_IP', Sequelize.col('ip')), 'ip'],
[Sequelize.fn('BIN_TO_UUID', Sequelize.col('uuid')), 'uuid'],
'country',
'cidr',
'org',
'pcheck',
],
where: { ip: ips.map((ip) => Sequelize.fn('IP_TO_BIN', ip)) },
raw: true,
});
result.forEach((obj) => {

View File

@ -4,25 +4,18 @@
*
*/
import { DataTypes } from 'sequelize';
import Sequelize, { DataTypes } from 'sequelize';
import sequelize from './sequelize';
import Channel from './Channel';
import RegUser from './RegUser';
const Message = sequelize.define('Message', {
// Message ID
id: {
type: DataTypes.INTEGER.UNSIGNED,
autoIncrement: true,
primaryKey: true,
},
name: {
type: `${DataTypes.CHAR(32)} CHARSET utf8mb4 COLLATE utf8mb4_unicode_ci`,
defaultValue: 'mx',
allowNull: false,
},
flag: {
type: DataTypes.CHAR(2),
defaultValue: 'xx',
@ -43,16 +36,57 @@ const Message = sequelize.define('Message', {
},
});
Message.belongsTo(Channel, {
as: 'channel',
foreignKey: 'cid',
onDelete: 'cascade',
});
export async function storeMessage(
flag,
message,
cid,
uid,
) {
await Channel.update({
lastMessage: Sequelize.literal('CURRENT_TIMESTAMP'),
}, {
where: {
id: cid,
},
});
return Message.create({
flag,
message,
cid,
uid,
});
}
Message.belongsTo(RegUser, {
as: 'user',
foreignKey: 'uid',
onDelete: 'cascade',
});
export async function getMessagesForChannel(cid, limit) {
const models = await Message.findAll({
attributes: [
'message',
'uid',
'flag',
[
Sequelize.fn('UNIX_TIMESTAMP', Sequelize.col('createdAt')),
'ts',
],
],
include: {
association: 'user',
attributes: ['name'],
},
where: { cid },
limit,
order: [['createdAt', 'DESC']],
raw: true,
});
return models.map((model) => {
const {
[user.name]: name,
message,
flag,
uid,
ts,
} = model;
return [name, message, flag, uid, ts];
});
}
export default Message;

View File

@ -1,5 +1,4 @@
/**
* Created by HF
*
* This is the database of the data for registered Users
*
@ -9,6 +8,9 @@ import Sequelize, { DataTypes, QueryTypes } from 'sequelize';
import sequelize from './sequelize';
import { generateHash } from '../../utils/hash';
import { USERLVL } from '../../core/constants';
export { USERLVL } from '../../core/constants';
const RegUser = sequelize.define('User', {
@ -21,6 +23,7 @@ const RegUser = sequelize.define('User', {
email: {
type: DataTypes.CHAR(40),
allowNull: true,
unique: true,
},
name: {
@ -38,22 +41,28 @@ const RegUser = sequelize.define('User', {
defaultValue: false,
},
// null if external oauth authentication
// null if only ever used external oauth
password: {
type: DataTypes.CHAR(60),
allowNull: true,
},
// currently just moderator
userlvl: {
type: DataTypes.TINYINT,
allowNull: false,
defaultValue: USERLVL.REGISTERED,
},
// currently just moderator TODO Delete
roles: {
type: DataTypes.TINYINT,
allowNull: false,
defaultValue: 0,
},
// mail and Minecraft verified
// if account is mail verified TODO Delete
verified: {
type: DataTypes.TINYINT,
type: DataTypes.BOOLEAN,
allowNull: false,
defaultValue: false,
},
@ -65,11 +74,13 @@ const RegUser = sequelize.define('User', {
defaultValue: 0,
},
// TODO Delete, replaced by ThreePID table
discordid: {
type: DataTypes.CHAR(20),
allowNull: true,
},
// TODO Delete, replaced by ThreePID table
redditid: {
type: DataTypes.CHAR(10),
allowNull: true,
@ -81,52 +92,22 @@ const RegUser = sequelize.define('User', {
type: DataTypes.DATE,
allowNull: true,
},
// flag == country code
flag: {
type: DataTypes.CHAR(2),
defaultValue: 'xx',
allowNull: false,
},
lastLogIn: {
type: DataTypes.DATE,
allowNull: true,
},
}, {
timestamps: true,
updatedAt: false,
getterMethods: {
mailVerified() {
return this.verified & 0x01;
},
blockDm() {
return this.blocks & 0x01;
},
isMod() {
return this.roles & 0x01;
},
},
setterMethods: {
mailVerified(num) {
const val = (num) ? (this.verified | 0x01) : (this.verified & ~0x01);
this.setDataValue('verified', val);
},
blockDm(num) {
const val = (num) ? (this.blocks | 0x01) : (this.blocks & ~0x01);
this.setDataValue('blocks', val);
},
isMod(num) {
const val = (num) ? (this.roles | 0x01) : (this.roles & ~0x01);
this.setDataValue('roles', val);
},
password(value) {
if (value) this.setDataValue('password', generateHash(value));
},
@ -192,10 +173,11 @@ export async function getNamesToIds(ids) {
}
/*
* take array of {id: useId, ...} object and resolve
* user information
* take array of objects that include user ids and add
* user informations if user is not private
* @param rawRanks array of {id: userId, ...} objects
*/
export async function populateRanking(rawRanks) {
export async function populateIdObj(rawRanks) {
if (!rawRanks.length) {
return rawRanks;
}
@ -219,15 +201,14 @@ export async function populateRanking(rawRanks) {
},
raw: true,
});
for (let i = 0; i < userData.length; i += 1) {
const { id, name, age } = userData[i];
for (const { id, name, age } of userData) {
const dat = rawRanks.find((r) => r.id === id);
if (dat) {
dat.name = name;
dat.age = age;
}
}
return rawRanks.filter((r) => r.name);
return rawRanks;
}
export default RegUser;

30
src/data/sql/ThreePID.js Normal file
View File

@ -0,0 +1,30 @@
/**
*
* Storing third party IDs for oauth login
*/
import { DataTypes } from 'sequelize';
import sequelize from './sequelize';
export { THREEP } from '../../core/constants';
const ThreePID = sequelize.define('ThreePID', {
id: {
type: DataTypes.INTEGER.UNSIGNED,
autoIncrement: true,
primaryKey: true,
},
provider: {
type: DataTypes.TINYINT,
allowNull: false,
},
tpid: {
type: DataTypes.CHAR(20),
allowNull: false,
},
});
export default ThreePID;

66
src/data/sql/UserIP.js Normal file
View File

@ -0,0 +1,66 @@
/*
*
* Last IP and useragent a user connected with
*
*/
import { DataTypes } from 'sequelize';
import sequelize from './sequelize';
const UserIP = sequelize.define('UserIP', {
ua: {
type: DataTypes.CHAR(200),
},
lastSeen: {
type: DataTypes.DATE,
defaultValue: DataTypes.NOW,
allowNull: false,
},
}, {
timestamps: false,
setterMethods: {
ua(value) {
const URIencode = encodeURIComponent(value);
this.setDataValue('ua', URIencode.slice(0, 200));
},
},
});
/*
* save lastIp of user
* @param uid userId
* @param ip IP
* @param ua UserAgent
*/
export function updateLastIp(uid, ip, ua) {
return UserIP.upsert({
uid,
ip,
ua,
lastSeen: new Date().toISOString(),
});
}
/*
* update lastSeen
*/
export async function touch(uid, ip, ua) {
try {
const data = {
lastSeen: new Date().toISOString(),
};
if (ua) {
data.ua = ua;
}
await UserIP.update(data, {
where: { uid, ip },
});
} catch {
return false;
}
return true;
}
export default UserIP;

126
src/data/sql/WhoisRange.js Normal file
View File

@ -0,0 +1,126 @@
import Sequelize, { DataTypes, QueryTypes } from 'sequelize';
import sequelize from './sequelize';
/*
* Information of IP Ranges from whois,
* min and max are the upper and lower bound of IPs within the range,
* stored in the same 64bit format as IP in IPInfo.js
*
* Will be kept indefinitelly, updated regularly
*/
const WhoisRange = sequelize.define('WhoisRange', {
id: {
type: DataTypes.INTEGER.UNSIGNED,
autoIncrement: true,
primaryKey: true,
},
min: {
type: 'VARBINARY(8)',
unique: true,
allowNull: false,
},
max: {
type: 'VARBINARY(8)',
unique: true,
allowNull: false,
},
mask: {
type: DataTypes.TINYINT.UNSIGNED,
allowNull: false,
},
country: {
type: DataTypes.CHAR(2),
defaultValue: 'xx',
allowNull: false,
},
org: {
type: `${DataTypes.STRING(60)} CHARSET utf8mb4 COLLATE utf8mb4_unicode_ci`,
set(value) {
this.setDataValue('org', value.slice(0, 60));
},
},
descr: {
type: `${DataTypes.STRING(60)} CHARSET utf8mb4 COLLATE utf8mb4_unicode_ci`,
set(value) {
this.setDataValue('descr', value.slice(0, 60));
},
},
asn: {
type: DataTypes.INTEGER.UNSIGNED,
},
checkedAt: {
type: DataTypes.DATE,
defaultValue: DataTypes.NOW,
allowNull: false,
},
}, {
timestamps: false,
});
/**
* Checks if range that includes ip exists,
* If it exists, updates the IPInfos table to associate the IP to this ranges
* Procedure got declared in ./sequelize.js
* @param ip as string
* @return {
* wid as number,
* cidr as string,
* asn as number,
* cuntry as two letter lowercase code,
* org as string,
* descr as string,
* } if exists or {
* host as string (whois host to query)
* } if whois host is known or null if not
*/
export async function getWhoisRangeOfIp(ip) {
try {
// return cidr, country, org, descr, asn, checkedAt
const rangeq = sequelize.query(
'CALL RANGE_OF_IP($1)',
{
bind: [ip],
type: QueryTypes.SELECT,
raw: true,
plain: true,
}
);
return rangeq;
} catch (err) {
console.error(`SQL Error on getRangeOfIp: ${err.message}`);
}
return null;
}
/**
* Save whois data to range
* @param whoisData
* @return id if successful, null if not
*/
export async function saveWhoisRange(whoisData) {
const { range, country, org, descr, asn } = whoisData;
try {
const [rangeq] = await WhoisRange.upsert({
min: Sequelize.fn('UNHEX', range[0]),
max: Sequelize.fn('UNHEX', range[1]),
mask: range[2],
country,
org,
descr,
asn,
checkedAt: new Date(),
});
return rangeq.id;
} catch (err) {
console.error(`SQL Error on saveIpRange: ${err.message}`);
}
return null;
}

View File

@ -0,0 +1,82 @@
import Sequelize, { DataTypes, QueryTypes } from 'sequelize';
import sequelize from './sequelize';
/*
* Information of whois hosts responsible for IPRanges,
* min and max are the upper and lower bound of IPs within the range,
* stored in the same 64bit format as IP in IPInfo.js
*
* Will be kept indefinitelly, updated regularly
*/
const WhoisReferral = sequelize.define('WhoisReferral', {
id: {
type: DataTypes.INTEGER.UNSIGNED,
autoIncrement: true,
primaryKey: true,
},
min: {
type: 'VARBINARY(8)',
unique: true,
allowNull: false,
},
max: {
type: 'VARBINARY(8)',
unique: true,
allowNull: false,
},
mask: {
type: DataTypes.TINYINT.UNSIGNED,
allowNull: false,
},
host: {
type: DataTypes.STRING(60),
allowNull: false,
},
checkedAt: {
type: DataTypes.DATE,
defaultValue: DataTypes.NOW,
allowNull: false,
},
}, {
timestamps: false,
});
export async function getWhoisReferraloOfIp(ip) {
try {
// return cidr, country, org, descr, asn, checkedAt
const rangeq = sequelize.query(
'CALL WHOIS_REFERRAL_OF_IP($1)',
{
bind: [ip],
type: QueryTypes.SELECT,
raw: true,
plain: true,
}
);
return rangeq?.host;
} catch (err) {
console.error(`SQL Error on getRangeOfIp: ${err.message}`);
}
return null;
}
export async function saveWhoisReferral(range, host) {
try {
const [rangeq] = await WhoisReferral.upsert({
min: Sequelize.fn('UNHEX', range[0]),
max: Sequelize.fn('UNHEX', range[1]),
mask: range[2],
host,
checkedAt: new Date(),
});
return rangeq.id;
} catch (err) {
console.error(`SQL Error on saveIpRange: ${err.message}`);
}
return null;
}

View File

@ -1,10 +1,14 @@
import sequelize from './sequelize';
import Whitelist from './Whitelist';
import RegUser from './RegUser';
import RegUser, { USERLVL } from './RegUser';
import Channel from './Channel';
import UserChannel from './UserChannel';
import Message from './Message';
import UserBlock from './UserBlock';
import IPInfo from './IPInfo';
import WhoisRange from './WhoisRange';
import UserIP from './UserIP';
import ThreePID, { THREEP } from './ThreePID';
/*
* User Channel access
@ -18,6 +22,54 @@ Channel.belongsToMany(RegUser, {
through: UserChannel,
});
/*
* ip informations of user
*/
IPInfo.belongsToMany(RegUser, {
through: UserIP,
foreignKey: 'ip',
});
RegUser.belongsToMany(IPInfo, {
through: UserIP,
foreignKey: 'uid',
});
/*
* third party ids for oauth login
*/
RegUser.hasMany(ThreePID, {
as: 'tp',
foreignKey: {
name: 'uid',
allowNull: false,
},
onDelete: 'CASCADE',
});
ThreePID.belongsTo(RegUser);
/*
* whois range for ip
*/
WhoisRange.hasMany(IPInfo, {
as: 'whois',
foreignKey: 'wid',
});
IPInfo.belongsTo(WhoisRange);
/*
* chat messages
*/
Message.belongsTo(Channel, {
as: 'channel',
foreignKey: 'cid',
onDelete: 'CASCADE',
});
Message.belongsTo(RegUser, {
as: 'user',
foreignKey: 'uid',
onDelete: 'CASCADE',
});
/*
* User blocks of other user
*
@ -35,12 +87,41 @@ RegUser.belongsToMany(RegUser, {
foreignKey: 'buid',
});
/*
* includes for RegUsert
* that should be available on ordinary
* login
*/
const regUserQueryInclude = [{
model: Channel,
as: 'channel',
include: [{
model: RegUser,
as: 'dmu1',
foreignKey: 'dmu1id',
attributes: ['id', 'name'],
}, {
model: RegUser,
as: 'dmu2',
foreignKey: 'dmu2id',
attributes: ['id', 'name'],
}],
}, {
association: 'blocked',
attributes: ['id', 'name'],
}];
export {
regUserQueryInclude,
Whitelist,
RegUser,
Channel,
UserChannel,
Message,
UserBlock,
WhoisRange,
IPInfo,
ThreePID,
USERLVL,
THREEP,
};

View File

@ -4,7 +4,6 @@
import Sequelize from 'sequelize';
import logger from '../../core/logger';
import {
MYSQL_HOST, MYSQL_DATABASE, MYSQL_USER, MYSQL_PW, LOG_MYSQL,
} from '../../core/config';
@ -18,11 +17,81 @@ const sequelize = new Sequelize(MYSQL_DATABASE, MYSQL_USER, MYSQL_PW, {
idle: 10000,
acquire: 10000,
},
logging: (LOG_MYSQL) ? (...msg) => logger.info(msg) : false,
// eslint-disable-next-line no-console
logging: (LOG_MYSQL) ? (...msg) => console.info(msg) : false,
dialectOptions: {
connectTimeout: 10000,
multipleStatements: true,
},
});
export const sync = async () => {
await sequelize.sync({ alter: { drop: true } });
/*
* custom functions (for IP_BIN explenation, look into IP_Info comments)
*/
const functions = {
IP_TO_BIN: `CREATE FUNCTION IF NOT EXISTS IP_TO_BIN(ip VARCHAR(39)) returns VARBINARY(8) DETERMINISTIC CONTAINS SQL
BEGIN
DECLARE longBin VARBINARY(16);
SET longBin = INET6_ATON(ip);
IF LENGTH(longBin) > 4
THEN
RETURN (cast(longBin as binary(8)));
ELSE
RETURN (longBin);
END IF;
END`,
BIN_TO_IP: `CREATE FUNCTION IF NOT EXISTS BIN_TO_IP(bin VARBINARY(8)) returns VARCHAR(21) DETERMINISTIC CONTAINS SQL
BEGIN
RETURN (INET6_NTOA(IF(LENGTH(bin) > 4, CAST(bin as BINARY(16)), bin)));
END`,
RANGE_OF_IP: `CREATE PROCEDURE IF NOT EXISTS RANGE_OF_IP(ip VARCHAR(39)) READS SQL DATA
BEGIN
DECLARE binIp VARBINARY(8);
SET binIp = IP_TO_BIN(ip);
SELECT id as wid, CONCAT(BIN_TO_IP(min), '/', mask) AS cidr, country, org, descr, asn FROM WhoisRanges WHERE min <= binIp AND max >= binIp AND LENGTH(binIP) = LENGTH(min) AND checkedAt > (NOW() - INTERVAL 1 MONTH) LIMIT 1;
IF FOUND_ROWS() = 0
THEN
SELECT host FROM WhoisReferrals WHERE min <= binIp AND max >= binIp AND LENGTH(binIp) = LENGTH(min) AND checkedAt > (NOW() - INTERVAL 1 MONTH) LIMIT 1;
END IF;
END`,
RANGE_OF_IP_OI: `CREATE PROCEDURE IF NOT EXISTS RANGE_OF_IP(ip VARCHAR(39)) READS SQL DATA
BEGIN
DECLARE binIp VARBINARY(8);
DECLARE q_id INTEGER UNSIGNED;
DECLARE q_cidr VARCHAR(22);
DECLARE q_country CHAR(2);
DECLARE q_org VARCHAR(60);
DECLARE q_descr VARCHAR(60);
DECLARE q_asn VARCHAR(12);
SET binIp = IP_TO_BIN(ip);
SELECT id, CONCAT(BIN_TO_IP(min), '/', mask), country, org, descr, asn FROM WhoisRanges WHERE min <= binIp AND max >= binIp AND LENGTH(binIP) = LENGTH(min) AND checkedAt > (NOW() - INTERVAL 1 MONTH) LIMIT 1 INTO q_id, q_cidr, q_country, q_org, q_descr, q_asn;
IF q_id IS NULL
THEN
SELECT host FROM WhoisReferrals WHERE min <= binIp AND max >= binIp AND LENGTH(binIp) = LENGTH(min) AND checkedAt > (NOW() - INTERVAL 1 MONTH) LIMIT 1;
ELSE
INSERT INTO IPInfos (ip, uuid, wid) VALUES (binIP, UUID_TO_BIN(UUID()), q_id) ON DUPLICATE KEY UPDATE wid = q_id;
SELECT q_cidr AS cidr, q_country AS country, q_org AS org, q_descr AS descr, q_asn AS asn;
END IF;
END`,
WHOIS_REFERRAL_OF_IP: `CREATE PROCEDURE IF NOT EXISTS WHOIS_REFERRAL_OF_IP(ip VARCHAR(39)) READS SQL DATA
BEGIN
DECLARE binIp VARBINARY(8);
SET binIp = IP_TO_BIN(ip);
SELECT host FROM WhoisReferrals WHERE min <= binIp AND max >= binIp AND LENGTH(binIP) = LENGTH(min);
END`,
};
for (const name of Object.keys(functions)) {
try {
// eslint-disable-next-line no-await-in-loop
await sequelize.query(functions[name], { raw: true });
} catch (err) {
console.error(`Error on creating SQL Function ${name}: ${err.message}`);
}
};
};
export default sequelize;

View File

@ -2,6 +2,7 @@ import express from 'express';
import logger from '../core/logger';
import { RegUser } from '../data/sql';
import { USERLVL } from '../core/constants';
import { getIPFromRequest } from '../utils/ip';
import { compareToHash } from '../utils/hash';
import { APISOCKET_KEY } from '../core/config';
@ -48,7 +49,7 @@ router.post('/checklogin', async (req, res) => {
'name',
'email',
'password',
'verified',
'userlvl',
],
};
let userString;
@ -94,14 +95,14 @@ router.post('/checklogin', async (req, res) => {
return;
}
logger.info(`ADMINAPI: User ${reguser.name} / ${reguser.id} got loged in`);
logger.info(`ADMINAPI: User ${reguser.name} / ${reguser.id} got logged in`);
res.json({
success: true,
userdata: {
id: reguser.id,
name: reguser.name,
email: reguser.email,
verified: !!reguser.verified,
verified: reguser.userlvl >= USERLVL.VERIFIED,
},
});
});

View File

@ -9,6 +9,7 @@ import { getHostFromRequest } from '../../../utils/ip';
import { compareToHash } from '../../../utils/hash';
import { checkIfMuted } from '../../../data/redis/chat';
import { checkIfMailDisposable } from '../../../core/isAllowed';
import { USERLVL } from '../../../core/constants';
async function validate(email, password, t, gettext) {
const errors = [];
@ -64,13 +65,18 @@ export default async (req, res) => {
return;
}
await user.regUser.update({
const { regUser } = user;
let { userlvl } = regUser;
if (userlvl <= USERLVL.VERIFIED && userlvl > USERLVL.REGISTERED) {
userlvl = USERLVL.REGISTERED;
}
await regUser.update({
email,
mailVerified: false,
userlvl,
});
const host = getHostFromRequest(req);
mailProvider.sendVerifyMail(email, user.regUser.name, host, lang);
mailProvider.sendVerifyMail(email, regUser.name, host, lang);
res.json({
success: true,

View File

@ -5,7 +5,7 @@ import { RegUser } from '../../../data/sql';
import mailProvider from '../../../core/MailProvider';
import getMe from '../../../core/me';
import { getIPFromRequest, getHostFromRequest } from '../../../utils/ip';
import { checkIfMailDisposable } from '../../../core/isAllowed';
import { checkIfMailDisposable } from '../../../core/ipUserIntel';
import {
validateEMail,
validateName,

View File

@ -4,6 +4,7 @@
import mailProvider from '../../../core/MailProvider';
import { getHostFromRequest } from '../../../utils/ip';
import { USERLVL } from '../../../data/sql';
export default async (req, res) => {
const { user, lang } = req;
@ -15,8 +16,8 @@ export default async (req, res) => {
return;
}
const { name, email, mailVerified } = user.regUser;
if (mailVerified) {
const { name, email, userlvl } = user.regUser;
if (userlvl >= USERLVL.VERIFIED) {
res.status(400);
res.json({
errors: ['You are already verified.'],

View File

@ -16,7 +16,7 @@ async function validate(email, gettext) {
}
export default async (req, res) => {
const ip = req.trueIp;
const { ip } = req.user;
const { email } = req.body;
const { gettext } = req.ttag;

View File

@ -31,9 +31,6 @@ async function block(req, res) {
if (!userName && !userId) {
errors.push('No userId or userName defined');
}
if (!user || !user.regUser) {
errors.push('You are not logged in');
}
if (user && userId && user.id === userId) {
errors.push('You can not block yourself.');
}

View File

@ -10,17 +10,9 @@ async function blockdm(req, res) {
const { block } = req.body;
const { user } = req;
const errors = [];
if (typeof block !== 'boolean') {
errors.push('Not defined if blocking or unblocking');
}
if (!user || !user.regUser) {
errors.push('You are not logged in');
}
if (errors.length) {
res.status(400);
res.json({
errors,
res.status(400).json({
errors: ['Not defined if blocking or unblocking'],
});
return;
}

View File

@ -1,5 +1,5 @@
/*
*
* tell user his own IID
*/
import {
getIPFromRequest,
@ -16,7 +16,6 @@ async function getiid(req, res, next) {
);
const iid = await getIIDofIP(ip);
if (!iid) {
throw new Error('Could not get IID');
}

View File

@ -2,9 +2,7 @@ import express from 'express';
import session from '../../core/session';
import passport from '../../core/passport';
import logger from '../../core/logger';
import User from '../../data/User';
import { getIPFromRequest } from '../../utils/ip';
import me from './me';
import auth from './auth';
@ -35,7 +33,6 @@ router.use(express.json());
// eslint-disable-next-line no-unused-vars
router.use((err, req, res, next) => {
logger.warn(`Got invalid json from ${req.trueIp} on ${req.originalUrl}`);
res.status(400).json({
errors: [{ msg: 'Invalid Request' }],
});
@ -54,8 +51,7 @@ router.use(session);
/*
* at this point we could use the session id to get
* stuff without having to verify the whole user,
* which would avoid SQL requests and it got used previously
* when we set pixels via api/pixel (new removed)
* which would avoid SQL requests
*/
/*
@ -69,22 +65,41 @@ router.use(passport.session());
/*
* modtools
* (does not json bodies, but urlencoded)
* (does not take urlencoded bodies)
*/
router.use('/modtools', modtools);
/*
* create dummy user with just ip if not
* logged in
* create unregistered user by request if
* not logged in
*/
router.use(async (req, res, next) => {
if (!req.user) {
req.user = new User();
await req.user.initialize(null, getIPFromRequest(req));
req.user = new User(req);
}
next();
});
router.get('/chathistory', chatHistory);
router.get('/me', me);
router.use('/auth', auth);
router.post('/banme', banme);
router.use((req, res, next) => {
if (!req.user.isRegistered) {
next(new Error('You are not logged in'));
return;
}
next();
});
/*
* require registered user after this point
*/
router.post('/startdm', startDm);
router.post('/leavechan', leaveChan);
@ -95,13 +110,9 @@ router.post('/blockdm', blockdm);
router.post('/privatize', privatize);
router.get('/chathistory', chatHistory);
router.get('/me', me);
router.post('/banme', banme);
router.use('/auth', auth);
/*
* error handling
*/
// eslint-disable-next-line no-unused-vars
router.use((err, req, res, next) => {

View File

@ -11,17 +11,9 @@ async function leaveChan(req, res) {
const channelId = parseInt(req.body.channelId, 10);
const { user } = req;
const errors = [];
if (channelId && Number.isNaN(channelId)) {
errors.push('Invalid channelId');
}
if (!user || !user.regUser) {
errors.push('You are not logged in');
}
if (errors.length) {
res.status(400);
res.json({
errors,
res.status(400).json({
errors: ['Invalid channelId'],
});
return;
}
@ -35,8 +27,7 @@ async function leaveChan(req, res) {
}
}
if (!channel) {
res.status(401);
res.json({
res.status(401).json({
errors: ['You are not in this channel'],
});
return;
@ -49,8 +40,7 @@ async function leaveChan(req, res) {
* Faction and Default channels should be impossible to leave
*/
if (channel.type !== 1) {
res.status(401);
res.json({
res.status(401).json({
errors: ['Can not leave this channel'],
});
return;

View File

@ -9,7 +9,7 @@ export default async (req, res, next) => {
try {
const { user, lang } = req;
const userdata = await getMe(user, lang);
user.updateLogInTimestamp();
user.touch();
res.json(userdata);
} catch (error) {
next(error);

View File

@ -9,7 +9,6 @@ import multer from 'multer';
import CanvasCleaner from '../../core/CanvasCleaner';
import chatProvider from '../../core/ChatProvider';
import { getIPFromRequest } from '../../utils/ip';
import { escapeMd } from '../../core/utils';
import logger, { modtoolsLogger } from '../../core/logger';
import {
@ -24,6 +23,7 @@ import {
removeMod,
makeMod,
} from '../../core/adminfunctions';
import { USERLVL } from '../../data/sql';
const router = express.Router();
@ -41,31 +41,22 @@ const upload = multer({
/*
* make sure User is logged in and mod or mod
* make sure User is logged in and at least Mod
*/
router.use(async (req, res, next) => {
const ip = getIPFromRequest(req);
if (!req.user) {
logger.warn(
`MODTOOLS: ${ip} tried to access modtools without login`,
);
const { t } = req.ttag;
res.status(403).send(t`You are not logged in`);
next(new Error(t`You are not logged in`));
return;
}
/*
* 1 = Admin
* 2 = Mod
*/
if (!req.user.userlvl) {
if (req.user.userlvl < USERLVL.MOD) {
logger.warn(
`MODTOOLS: ${ip} / ${req.user.id} tried to access modtools`,
`MODTOOLS: ${req.user.ip} / ${req.user.id} tried to access modtools`,
);
const { t } = req.ttag;
res.status(403).send(t`You are not allowed to access this page`);
next(new Error(t`You are not allowed to access this page`));
return;
}
next();
});
@ -77,7 +68,7 @@ router.post('/', upload.single('image'), async (req, res, next) => {
const aLogger = (text) => {
const timeString = new Date().toLocaleTimeString();
// eslint-disable-next-line max-len
const logText = `@[${escapeMd(req.user.regUser.name)}](${req.user.id}) ${text}`;
const logText = `@[${escapeMd(req.user.name)}](${req.user.id}) ${text}`;
modtoolsLogger.info(
`${timeString} | MODTOOLS> ${logText}`,
);
@ -90,7 +81,7 @@ router.post('/', upload.single('image'), async (req, res, next) => {
};
const bLogger = (text) => {
logger.info(`IID> ${req.user.regUser.name}[${req.user.id}]> ${text}`);
logger.info(`IID> ${req.user.name}[${req.user.id}]> ${text}`);
};
try {
@ -186,7 +177,7 @@ router.post('/', upload.single('image'), async (req, res, next) => {
brcoor,
canvasid,
aLogger,
(req.user.userlvl === 1),
(req.user.userlvl >= USERLVL.ADMIN),
);
res.status(ret).send(msg);
return;
@ -202,7 +193,7 @@ router.post('/', upload.single('image'), async (req, res, next) => {
* just admins past here, no Mods
*/
router.use(async (req, res, next) => {
if (req.user.userlvl !== 1) {
if (req.user.userlvl < USERLVL.ADMIN) {
const { t } = req.ttag;
res.status(403).send(t`Just admins can do that`);
return;
@ -215,7 +206,7 @@ router.use(async (req, res, next) => {
*/
router.post('/', async (req, res, next) => {
const aLogger = (text) => {
logger.info(`ADMIN> ${req.user.regUser.name}[${req.user.id}]> ${text}`);
logger.info(`ADMIN> ${req.user.name}[${req.user.id}]> ${text}`);
};
try {
@ -251,8 +242,8 @@ router.post('/', async (req, res, next) => {
}
});
router.use(async (req, res) => {
res.status(400).send('Invalid request');
router.use(async (req, res, next) => {
next(new Error('Invalid request'));
});
// eslint-disable-next-line no-unused-vars

View File

@ -9,17 +9,9 @@ async function privatize(req, res) {
const { priv } = req.body;
const { user } = req;
const errors = [];
if (typeof priv !== 'boolean') {
errors.push('Not defined if setting or unsetting private');
}
if (!user || !user.regUser) {
errors.push('You are not logged in');
}
if (errors.length) {
res.status(400);
res.json({
errors,
res.status(400).json({
errors: ['Not defined if setting or unsetting private'],
});
return;
}

View File

@ -27,10 +27,7 @@ async function startDm(req, res) {
if (!userName && !userId) {
errors.push('No userId or userName defined');
}
if (!user || !user.regUser) {
errors.push('You are not logged in');
}
if (user && userId && user.id === userId) {
if (userId && user.id === userId) {
errors.push('You can not DM yourself.');
}
if (errors.length) {
@ -73,7 +70,7 @@ async function startDm(req, res) {
}
logger.info(
`Creating DM Channel between ${user.regUser.name} and ${userName}`,
`Creating DM Channel between ${user.name} and ${userName}`,
);
/*
* start DM session

View File

@ -10,7 +10,7 @@ import http from 'http';
import forceGC from './core/forceGC';
import logger from './core/logger';
import rankings from './core/Ranks';
import sequelize from './data/sql/sequelize';
import { sync as syncSql } from './data/sql/sequelize';
import { connect as connectRedis } from './data/redis/client';
import routes from './routes';
import chatProvider from './core/ChatProvider';
@ -84,7 +84,7 @@ app.use(routes);
// ip config
// -----------------------------------------------------------------------------
// sync sql models
sequelize.sync({ alter: { drop: false } })
syncSql()
// connect to redis
.then(connectRedis)
.then(async () => {
@ -122,7 +122,7 @@ sequelize.sync({ alter: { drop: false } })
* initializers that rely on the cluster being fully established
* i.e. to know if it is the shard that runs the event
*/
if (socketEvents.isCluster && socketEvents.amIImportant()) {
if (socketEvents.isCluster && socketEvents.important) {
logger.info('I am the main shard');
}
rankings.initialize();

View File

@ -41,10 +41,11 @@ const LISTEN_PREFIX = 'l';
class MessageBroker extends SocketEvents {
isCluster = true;
thisShard = SHARD_NAME;
constructor() {
super();
this.isCluster = true;
this.thisShard = SHARD_NAME;
/*
* currently running cross-shard requests,
* are tracked in order to only send them to receiving
@ -75,6 +76,28 @@ class MessageBroker extends SocketEvents {
setInterval(this.checkHealth, 10000);
}
get important() {
/*
* important main shard does tasks like running RpgEvent
* or updating rankings
*/
return !this.shardOnlineCounters[0]
|| this.shardOnlineCounters[0][0] === this.thisShard;
}
get lowestActiveShard() {
let lowest = 0;
let lShard = null;
this.shardOnlineCounters.forEach((shardData) => {
const [shard, cnt] = shardData;
if (cnt.total < lowest || !lShard) {
lShard = shard;
lowest = cnt.total;
}
});
return lShard || this.thisShard;
}
async initialize() {
this.publisher = pubsub.publisher;
this.subscriber = pubsub.subscriber;
@ -96,7 +119,7 @@ class MessageBroker extends SocketEvents {
console.log('CLUSTER: Initialized message broker');
}
/*
/**
* messages on shared broadcast channels that every shard is listening to
*/
async onShardBCMessage(message) {
@ -129,7 +152,7 @@ class MessageBroker extends SocketEvents {
super.emit(key, ...val);
return;
}
/*
/**
* other messages are shard names that announce the existence
* of a shard
*/
@ -151,7 +174,7 @@ class MessageBroker extends SocketEvents {
}
}
/*
/**
* messages on shard specific listener channel
* messages in form `type,JSONArrayData`
* straight emitted as socket event
@ -168,29 +191,7 @@ class MessageBroker extends SocketEvents {
}
}
getLowestActiveShard() {
let lowest = 0;
let lShard = null;
this.shardOnlineCounters.forEach((shardData) => {
const [shard, cnt] = shardData;
if (cnt.total < lowest || !lShard) {
lShard = shard;
lowest = cnt.total;
}
});
return lShard || this.thisShard;
}
amIImportant() {
/*
* important main shard does tasks like running RpgEvent
* or updating rankings
*/
return !this.shardOnlineCounters[0]
|| this.shardOnlineCounters[0][0] === this.thisShard;
}
/*
/**
* requests that go over all shards and combine responses from all
*/
req(type, ...args) {
@ -256,7 +257,7 @@ class MessageBroker extends SocketEvents {
this.sumOnlineCounters();
}
/*
/**
* messages on binary shard channels, where specific shards send from
*/
onShardBinaryMessage(buffer, shard) {
@ -314,7 +315,7 @@ class MessageBroker extends SocketEvents {
this.publisher.publish(BROADCAST_CHAN, msg);
}
/*
/**
* broadcast pixel message via websocket
* @param canvasId number ident of canvas
* @param chunkid number id consisting of i,j chunk coordinates
@ -337,7 +338,7 @@ class MessageBroker extends SocketEvents {
}
setCoolDownFactor(fac) {
if (this.amIImportant()) {
if (this.important) {
this.emit('setCoolDownFactor', fac);
} else {
super.emit('setCoolDownFactor', fac);

View File

@ -8,6 +8,11 @@ import {
} from './packets/server';
class SocketEvents extends EventEmitter {
isCluster = false;
// object with amount of online users
// in total and per canvas
onlineCounter;
constructor() {
super();
/*
@ -23,18 +28,18 @@ class SocketEvents extends EventEmitter {
}
// eslint-disable-next-line class-methods-use-this
async initialize() {
// nothing, only for child classes
get important() {
return true;
}
// eslint-disable-next-line class-methods-use-this
getLowestActiveShard() {
get lowestActiveShard() {
return null;
}
// eslint-disable-next-line class-methods-use-this
amIImportant() {
return true;
async initialize() {
// nothing, only for child classes
}
/*
@ -48,7 +53,7 @@ class SocketEvents extends EventEmitter {
});
}
/*
/**
* requests that expect a response
* req(type, args) can be awaited
* it will return a response from whatever listens on onReq(type, cb(args))
@ -84,7 +89,7 @@ class SocketEvents extends EventEmitter {
});
}
/*
/**
* broadcast pixel message via websocket
* @param canvasId number ident of canvas
* @param chunkid number id consisting of i,j chunk coordinates
@ -102,7 +107,7 @@ class SocketEvents extends EventEmitter {
this.emit('chunkUpdate', canvasId, [i, j]);
}
/*
/**
* chunk updates from event, image upload, etc.
* everything that's not a pixelUpdate and changes chunks
* @param canvasId
@ -115,7 +120,26 @@ class SocketEvents extends EventEmitter {
this.emit('chunkUpdate', canvasId, chunk);
}
/*
/**
* broadcast fetched IpInfo of user,
* used to spread flag informations of users
* to shards
* @param userIpInfo object {
* userId, // 0 if not logged in
* status, // proxycheck and ban status (see core/isAllowed)
* ip,
* cidr,
* org,
* country,
* asn,
* descr,
* }
*/
gotUserIpInfo(userIpInfo) {
this.emit('useripinfo', userIpInfo);
}
/**
* ask other shards to send email for us,
* only used when USE_MAILER is false
* @param type type of mail to send
@ -125,7 +149,7 @@ class SocketEvents extends EventEmitter {
this.emit('mail', ...args);
}
/*
/**
* received Chat message on own websocket
* @param user User Instance that sent the message
* @param message text message
@ -139,7 +163,7 @@ class SocketEvents extends EventEmitter {
this.emit('recvChatMessage', user, message, channelId);
}
/*
/**
* set cooldownfactor
* (used by RpgEvent)
* @param fac factor by which cooldown changes globally
@ -148,7 +172,7 @@ class SocketEvents extends EventEmitter {
this.emit('setCoolDownFactor', fac);
}
/*
/**
* broadcast chat message to all users in channel
* @param name chatname
* @param message Message to send
@ -174,7 +198,7 @@ class SocketEvents extends EventEmitter {
);
}
/*
/**
* send chat message to a single user in channel
*/
broadcastSUChatMessage(
@ -196,7 +220,7 @@ class SocketEvents extends EventEmitter {
);
}
/*
/**
* broadcast Assigning chat channel to user
* @param userId numerical id of user
* @param channelId numerical id of chat channel
@ -215,7 +239,7 @@ class SocketEvents extends EventEmitter {
);
}
/*
/**
* broadcast Removing chat channel from user
* @param userId numerical id of user
* @param channelId numerical id of chat channel
@ -228,7 +252,7 @@ class SocketEvents extends EventEmitter {
this.emit('remChatChannel', userId, channelId);
}
/*
/**
* trigger rate limit of ip
* @param ip
* @param blockTime in ms
@ -237,7 +261,7 @@ class SocketEvents extends EventEmitter {
this.emit('rateLimitTrigger', ip, blockTime);
}
/*
/**
* broadcast ranking list updates
* @param {
* dailyRanking?: daily pixel raking top 100,
@ -249,14 +273,14 @@ class SocketEvents extends EventEmitter {
this.emit('rankingListUpdate', rankings);
}
/*
/**
* reload user on websocket to get changes
*/
reloadUser(name) {
this.emit('reloadUser', name);
}
/*
/**
* broadcast online counter
* @param online Object of total and canvas online users
* (see this.onlineCounter)

View File

@ -33,8 +33,8 @@ import socketEvents from './socketEvents';
import chatProvider, { ChatProvider } from '../core/ChatProvider';
import authenticateClient from './authenticateClient';
import drawByOffsets from '../core/draw';
import isIPAllowed from '../core/isAllowed';
import { HOUR } from '../core/constants';
import isIPAllowed from '../core/ipUserIntel';
import { checkCaptchaSolution } from '../data/redis/captcha';
@ -106,6 +106,10 @@ class SocketServer {
});
});
socketEvents.on('useripinfo', (userIpInfo) => {
});
socketEvents.on('onlineCounter', (online) => {
const onlineBuffer = dehydrateOnlineCounter(online);
this.broadcast(onlineBuffer);
@ -188,7 +192,7 @@ class SocketServer {
async handleUpgrade(request, socket, head) {
const { headers } = request;
const ip = getIPFromRequest(request);
// trigger proxycheck
// trigger proxycheck TODO DO WE REMOVE THIS????
isIPAllowed(ip);
/*
* rate limit
@ -235,6 +239,7 @@ class SocketServer {
socket.destroy();
return;
}
user.isAllowed();
request.user = user;
this.wss.handleUpgrade(request, socket, head, (ws) => {
this.wss.emit('connection', ws, request);
@ -329,8 +334,9 @@ class SocketServer {
killAllWsByUerIp(ip) {
let amount = ipCounter.get(ip);
if (!amount) return 0;
if (!amount) {
return 0;
}
for (const [chunkid, clients] of this.CHUNK_CLIENTS.entries()) {
const newClients = clients.filter((ws) => ws.user.ip !== ip);
if (clients.length !== newClients.length) {
@ -630,18 +636,23 @@ class SocketServer {
if (!clients) {
clients = [];
this.CHUNK_CLIENTS.set(chunkid, clients);
} else if (clients.includes(ws)) {
return;
}
const pos = clients.indexOf(ws);
if (~pos) return;
clients.push(ws);
}
deleteChunk(chunkid, ws) {
ws.chunkCnt -= 1;
if (!this.CHUNK_CLIENTS.has(chunkid)) return;
const clients = this.CHUNK_CLIENTS.get(chunkid);
if (!clients) {
return;
}
const pos = clients.indexOf(ws);
if (~pos) clients.splice(pos, 1);
if (pos === -1) {
return;
}
ws.chunkCnt -= 1;
clients.splice(pos, 1);
}
deleteAllChunks(ws) {
@ -650,10 +661,12 @@ class SocketServer {
}
for (const client of this.CHUNK_CLIENTS.values()) {
const pos = client.indexOf(ws);
if (~pos) {
if (pos !== -1) {
client.splice(pos, 1);
ws.chunkCnt -= 1;
if (!ws.chunkCnt) return;
if (!ws.chunkCnt) {
return;
}
}
}
}

View File

@ -8,7 +8,6 @@ import session from '../core/session';
import passport from '../core/passport';
import User from '../data/User';
import { expressTTag } from '../core/ttag';
import { getIPFromRequest } from '../utils/ip';
const router = express.Router();
@ -24,16 +23,12 @@ function authenticateClient(req) {
return new Promise(
((resolve) => {
router(req, {}, async () => {
const country = req.headers['cf-ipcountry'] || 'xx';
const countryCode = country.toLowerCase();
let user;
if (req.user) {
user = req.user;
} else {
user = new User();
await user.initialize(null, getIPFromRequest(req));
user = new User(req);
}
user.setCountry(countryCode);
user.ttag = req.ttag;
user.lang = req.lang;
resolve(user);

View File

@ -45,7 +45,7 @@ function generateMainPage(req) {
const ssvR = {
...ssv,
shard: (host.startsWith(`${socketEvents.thisShard}.`))
? null : socketEvents.getLowestActiveShard(),
? null : socketEvents.lowestActiveShard,
lang: lang === 'default' ? 'en' : lang,
};
const scripts = (assets[`client-${lang}`])

View File

@ -41,7 +41,7 @@ function generatePopUpPage(req) {
const ssvR = {
...ssv,
shard: (host.startsWith(`${socketEvents.thisShard}.`))
? null : socketEvents.getLowestActiveShard(),
? null : socketEvents.lowestActiveShard,
lang: lang === 'default' ? 'en' : lang,
};
const script = (assets[`popup-${lang}`])

View File

@ -1,9 +1,12 @@
import { USERLVL } from '../../core/constants';
const initialState = {
id: null,
name: null,
wait: null,
coolDown: null, // ms
lastCoolDownEnd: null,
userlvl: USERLVL.ANONYM,
// messages are sent by api/me, like not_verified status
messages: [],
mailreg: false,
@ -15,8 +18,6 @@ const initialState = {
isOnMobile: false,
// small notifications for received cooldown
notification: null,
// 1: Admin, 2: Mod, 0: ordinary user
userlvl: 0,
};
export default function user(
@ -103,7 +104,7 @@ export default function user(
mailreg: false,
blockDm: false,
priv: false,
userlvl: 0,
userlvl: USERLVL.ANONYM,
};
}

View File

@ -3,7 +3,11 @@
*
*/
import { TILE_ZOOM_LEVEL, TILE_SIZE } from '../core/constants';
import {
TILE_ZOOM_LEVEL,
TILE_SIZE,
BACKGROUND_CLR_HEX,
} from '../core/constants';
import {
getTileOfPixel,
@ -22,6 +26,8 @@ import Renderer from './Renderer';
import ChunkLoader from './ChunkLoader2D';
import pixelNotify from './PixelNotify';
const LOG_FRAC = Math.log2(TILE_ZOOM_LEVEL);
class Renderer2D extends Renderer {
is3D = false;
//
@ -178,13 +184,12 @@ class Renderer2D extends Renderer {
pixelNotify.updateScale(viewscale);
let tiledScale = (viewscale > 0.5)
? 0
: Math.round(Math.log2(viewscale) * 2 / TILE_ZOOM_LEVEL);
: Math.round(Math.log2(viewscale) / LOG_FRAC);
tiledScale = TILE_ZOOM_LEVEL ** tiledScale;
const canvasMaxTiledZoom = (isHistoricalView)
? this.historicalCanvasMaxTiledZoom
: this.canvasMaxTiledZoom;
const tiledZoom = canvasMaxTiledZoom + Math.log2(tiledScale)
* 2 / TILE_ZOOM_LEVEL;
const tiledZoom = canvasMaxTiledZoom + Math.log2(tiledScale) / LOG_FRAC;
const relScale = viewscale / tiledScale;
this.tiledScale = tiledScale;
@ -345,7 +350,7 @@ class Renderer2D extends Renderer {
if (scale > this.scaleThreshold) relScale = 1.0;
// scale
context.save();
context.fillStyle = '#C4C4C4';
context.fillStyle = BACKGROUND_CLR_HEX;
context.scale(relScale, relScale);
// decide if we update the timestamps of accessed chunks
const curTime = Date.now();
@ -576,7 +581,7 @@ class Renderer2D extends Renderer {
const CHUNK_RENDER_RADIUS_Y = Math.ceil(height / TILE_SIZE / 2 / scale);
context.save();
context.fillStyle = '#C4C4C4';
context.fillStyle = BACKGROUND_CLR_HEX;
// clear canvas and do nothing if no time selected
if (!historicalDate || !historicalTime) {
context.fillRect(0, 0, this.canvas.width, this.canvas.height);

View File

@ -16,6 +16,7 @@ import {
} from '../core/utils';
import {
THREE_TILE_SIZE,
BACKGROUND_CLR_HEX,
} from '../core/constants';
import {
setHover,
@ -145,7 +146,7 @@ class Renderer3D extends Renderer {
oobGeometry.rotateX(-Math.PI / 2);
oobGeometry.translate(THREE_TILE_SIZE / 2, 0.2, THREE_TILE_SIZE / 2);
const oobMaterial = new THREE.MeshLambertMaterial({
color: '#C4C4C4',
color: BACKGROUND_CLR_HEX,
});
this.oobGeometry = oobGeometry;
this.oobMaterial = oobMaterial;

View File

@ -129,14 +129,12 @@ class PcKeyProvider {
}
const queriesToday = Number(usage['Queries Today']) || 0;
const availableBurst = Number(usage['Burst Tokens Available']) || 0;
const burstActive = Number(usage['Burst Token Active']) === 1;
let dailyLimit = Number(usage['Daily Limit']) || 0;
let burstActive = false;
let availableQueries = dailyLimit - queriesToday;
if (availableQueries < 0) {
burstActive = true;
if (burstActive) {
dailyLimit *= 5;
availableQueries = dailyLimit - queriesToday;
}
const availableQueries = dailyLimit - queriesToday;
// eslint-disable-next-line max-len
this.logger.info(`PCKey: ${key}, Queries Today: ${availableQueries} / ${dailyLimit} (Burst: ${availableBurst}, ${burstActive ? 'active' : 'inactive'})`);
const keyData = [
@ -146,7 +144,7 @@ class PcKeyProvider {
availableBurst,
false,
];
if (burstActive || availableQueries > HYSTERESIS) {
if (availableQueries > HYSTERESIS) {
/*
* data is a few minutes old, stop at HYSTERESIS
*/
@ -157,7 +155,6 @@ class PcKeyProvider {
}
/*
* TODO: proxycheck added the used burst token to API
* query the API for limits
* @param key
*/
@ -411,8 +408,6 @@ class ProxyCheck {
/*
* same as for ip
* TODO: cache for mail providers, remember
* a disposable provider for an hour or so
* @param email
* @return Promise that resolves to
* null: failure

View File

@ -1,5 +1,5 @@
/*
* set CORS Headers
* express middleware to set CORS Headers
*/
import { CORS_HOSTS } from '../core/config';
@ -31,6 +31,7 @@ export default (req, res, next) => {
res.set({
'Access-Control-Allow-Headers': 'Content-Type',
'Access-Control-Allow-Methods': 'GET,POST',
'Access-Control-Max-Age': '86400',
});
res.sendStatus(200);
return;

View File

@ -1,49 +1,9 @@
/**
*
* basic functions to get data from headers and parse IPs
* basic functions for parsing IPs
*/
import { USE_XREALIP } from '../core/config';
/*
* Parse ip4 string to 32bit integer
* @param ipString ip string
* @return ipNum numerical ip
*/
function ip4ToNum(ipString) {
if (!ipString) {
return null;
}
const ipArr = ipString
.trim()
.split('.')
.map((numString) => parseInt(numString, 10));
if (ipArr.length !== 4 || ipArr.some(
(num) => Number.isNaN(num) || num > 255 || num < 0,
)) {
return null;
}
return (ipArr[0] << 24)
+ (ipArr[1] << 16)
+ (ipArr[2] << 8)
+ ipArr[3];
}
/*
* Parse ip4 number to string representation
* @param ipNum numerical ip (32bit integer)
* @return ipString string representation of ip (xxx.xxx.xxx.xxx)
*/
function ip4NumToStr(ipNum) {
return [
ipNum >>> 24,
ipNum >>> 16 & 0xFF,
ipNum >>> 8 & 0xFF,
ipNum & 0xFF,
].join('.');
}
/*
/**
* Get hostname from request
* @param req express req object
* @param includeProto if we include protocol (https, http)
@ -64,37 +24,11 @@ export function getHostFromRequest(req, includeProto = true, stripSub = false) {
if (!includeProto) {
return host;
}
const proto = headers['x-forwarded-proto'] || 'http';
return `${proto}://${host}`;
}
/*
* Get IP from request
* @param req express req object
* @return ip as string
*/
export function getIPFromRequest(req) {
if (USE_XREALIP) {
const ip = req.headers['x-real-ip'];
if (ip) {
return ip;
}
}
const { socket, connection } = req;
let conip = (connection ? connection.remoteAddress : socket.remoteAddress);
conip = conip || '0.0.0.1';
if (!USE_XREALIP) {
// eslint-disable-next-line no-console
console.warn(
`Connection not going through reverse proxy! IP: ${conip}`, req.headers,
);
}
return conip;
}
/*
/**
* Check if IP is v6 or v4
* @param ip ip as string
* @return true if ipv6, false otherwise
@ -103,46 +37,129 @@ export function isIPv6(ip) {
return ip.includes(':');
}
/*
* Set last digits of IPv6 to zero,
* needed because IPv6 assigns subnets to customers, we don't want to
* mess with individual ips
* @param ip ip as string (v4 or v6)
* @return ip as string, and if v6, the last digits set to 0
/**
* unpack IPv6 address into 8 blocks
* @param ip IPv6 IP string
* @return Array with length 8 of IPv6 parts as strings
*/
export function getIPv6Subnet(ip) {
export function unpackIPv6(ip) {
let ipUnpack = ip.split(':');
const spacer = ipUnpack.indexOf('');
if (~spacer) {
ipUnpack = ipUnpack.filter((a) => a);
ipUnpack.splice(spacer, 0, ...Array(8 - ipUnpack.length).fill('0'));
}
return ipUnpack;
}
/**
* Get hex representation of IP
* @param ip ip as string (if IPv6, the first 64bit have to be unpacked)
* @return hex string (without leading '0x')
*/
export function ipToHex(ip) {
if (isIPv6(ip)) {
// eslint-disable-next-line max-len
return `${ip.split(':')
return ip.split(':')
.slice(0, 4)
.join(':')}:0000:0000:0000:0000`;
.map((n) => `000${n}`.slice(-4).toLowerCase())
.join('');
}
return ip.split('.')
.map((n) => `0${parseInt(n, 10).toString(16)}`.slice(-2))
.join('');
}
/**
* Parse IPv4 string to Number and IPv6 string to 2xNumbers (first 64bit)
* @param ipString ip string
* @return numerical IP (Number or BigInt)
*/
function ipToNum(ipString) {
if (!ipString) {
return null;
}
if (isIPv6(ipString)) {
// IPv6
const hex = unpackIPv6(ipString.trim())
.map((n) => `000${n}`.slice(-4))
.slice(0, 4).join('');
try {
return BigInt(`0x${hex}`);
} catch {
return null;
}
}
// IPv4
const ipArr = ipString
.split('.')
.map((numString) => parseInt(numString, 10));
if (ipArr.length !== 4 || ipArr.some((num) => Number.isNaN(num))) {
return null;
}
// >>>0 is needed to convert from signed to unsigned
return ((ipArr[0] << 24)
+ (ipArr[1] << 16)
+ (ipArr[2] << 8)
+ ipArr[3])>>>0;
}
/**
* parse num representation of IP to hex
* @param num IP as 64bit BigInt if IPv6, Number if IPv4
* @return hex
*/
function numToHex(num) {
let ip = `00000000${num.toString(16)}`;
return ip.slice(
(typeof num === 'bigint') ? -16 : -8,
);
}
/**
* parse range into readable string
* @param range [start, end, mask] with start and end in hex
* @return string
*/
export function rangeToString(range) {
if (!range) {
return undefined;
}
return `${hexToIp(range[0])}/${range[2]}`;
}
/**
* parse hex representation of IP to string
* @param hex IP as 64bit hex for IPv6 and 32bit hex for IPv4
* @return ip as string
*/
export function hexToIp(hex) {
let ip = '';
if (hex.length === 8) {
// IPv4
let i = 0;
// eslint-disable-next-line no-constant-condition
while (true) {
ip += parseInt(hex.slice(i, i += 2), 16).toString(10);
if (i >= 8) break;
ip += '.';
}
} else {
// IPv6
for (let i = 0; i < 16; i += 4) {
ip += hex.slice(i, i + 4);
ip += ':';
}
ip += ':';
}
return ip;
}
/*
* Get numerical start and end of range
* @param range string of range in the format 'xxx.xxx.xxx.xxx - xxx.xxx.xxx.xxx'
* @return [start, end] with numerical IPs (32bit integer)
*/
function ip4RangeStrToRangeNum(range) {
if (!range) {
return null;
}
const [start, end] = range.split('-')
.map(ip4ToNum);
if (!start || !end || start > end) {
return null;
}
return [start, end];
}
/*
* Get Array of CIDRs for an numerical IPv4 range
* @param [start, end] with numerical IPs (32bit integer)
/**
* Get Array of CIDRs for an 32bit numerical IP range
* @param [start, end] with numerical IPs as Number
* @return Array of CIDR strings
*/
function ip4RangeNumToCIDR([start, end]) {
function ip32RangeNumToCIDR(start, end, ip) {
let maskNum = 32;
let mask = 0xFFFFFFFF;
const diff = start ^ end;
@ -151,62 +168,114 @@ function ip4RangeNumToCIDR([start, end]) {
maskNum -= 1;
}
if ((start & (~mask)) || (~(end | mask))) {
const divider = start | (~mask >> 1);
return ip4RangeNumToCIDR([start, divider]).concat(
ip4RangeNumToCIDR([divider + 1, end]),
const divider = (start | (~mask >> 1))>>>0;
if (ip) {
if (ip <= divider) {
return ip32RangeNumToCIDR(start, divider, ip);
}
return ip32RangeNumToCIDR(divider + 1, end, ip);
}
return ip32RangeNumToCIDR(start, divider).concat(
ip32RangeNumToCIDR(divider + 1, end),
);
}
return [`${ip4NumToStr(start)}/${maskNum}`];
return [[start, end, maskNum]];
}
/*
* Get Array of CIDRs for an IPv4 range
* @param range string of range in the format 'xxx.xxx.xxx.xxx - xxx.xxx.xxx.xxx'
/**
* Get Array of CIDRs for an 64bit numerical IP range
* @param [start, end] with numerical IPs as BigInt
* @param ip if given, return only range that includes ip
* @return Array of CIDR strings
*/
export function ip4RangeToCIDR(range) {
const rangeNum = ip4RangeStrToRangeNum(range);
if (!rangeNum) {
return null;
}
return ip4RangeNumToCIDR(rangeNum);
}
/*
* Get specific CIDR in numeric range that includes numeric ip
* @param ip numerical ip (32bit integer)
* @param [start, end] with numerical IPs (32bit integer)
* @return CIDR string
*/
function ip4NumInRangeNumToCIDR(ip, [start, end]) {
let maskNum = 32;
let mask = 0xFFFFFFFF;
function ip64RangeNumToCIDR(start, end, ip) {
let maskNum = 64;
const mask64 = 0xFFFFFFFFFFFFFFFFn;
let mask = mask64;
const diff = start ^ end;
while (diff & mask) {
mask <<= 1;
mask = (mask << 1n) & mask64;
maskNum -= 1;
}
if ((start & (~mask)) || (~(end | mask))) {
const divider = start | (~mask >> 1);
if (ip <= divider) {
return ip4NumInRangeNumToCIDR(ip, [start, divider]);
const invMask = ~mask & mask64;
if ((start & invMask) || (~(end | mask) & mask64)) {
const divider = start | (invMask >> 1n);
if (ip) {
if (ip <= divider) {
return ip64RangeNumToCIDR(start, divider, ip);
}
return ip64RangeNumToCIDR(divider + 1n, end, ip);
}
return ip4NumInRangeNumToCIDR(ip, [divider + 1, end]);
return ip64RangeNumToCIDR(start, divider).concat(
ip64RangeNumToCIDR(divider + 1n, end),
);
}
return `${ip4NumToStr(start)}/${maskNum}`;
return [[start, end, maskNum]];
}
/*
* Get specific CIDR in range that includes ip
* @param ip ip string ('xxx.xxx.xxx.xxx')
* @param range string ('xxx.xxx.xxx.xxx - xxx.xxx.xxx.xxx')
* @return CIDR string
/**
* Parse subnet given as string into array numerical representations
* @param subnet given as CIDR or range
* @param ip if given, return only range that includes ip
* @return [start, end, mask] start and end as hex and mask part of CIDR
* Array of same if ip isn't given and there could be multiple
*/
export function ip4InRangeToCIDR(ip, range) {
const rangeNum = ip4RangeStrToRangeNum(range);
const ipNum = ip4ToNum(ip);
if (!ipNum || !rangeNum || rangeNum[0] > ip || rangeNum[1] < ip) {
export function ipSubnetToHex(subnet, ip) {
const ipNum = ip && ipToNum(ip);
if (!subnet || (ip && !ipNum)) {
return null;
}
return ip4NumInRangeNumToCIDR(ipNum, rangeNum);
let ranges;
if (subnet.includes('-')) {
// given as range
const [start, end] = subnet.split('-').map(ipToNum);
if (!start
|| typeof start !== typeof end
|| start > end
|| (ipNum && typeof ipNum !== typeof start)
) {
return null;
}
const numRanges = (typeof start === 'bigint')
? ip64RangeNumToCIDR(start, end, ipNum)
: ip32RangeNumToCIDR(start, end, ipNum);
ranges = numRanges;
} else {
// given as CIDR
let [start, mask] = subnet.split('/');
start = ipToNum(start);
mask = parseInt(mask, 10);
if (!start || !mask) {
return null;
}
let end;
if (typeof start === 'bigint') {
// IPv6
if (mask >= 64) {
end = start;
} else {
const bitmask = (0xFFFFFFFFFFFFFFFFn >> BigInt(mask));
start = start & (~bitmask & 0xFFFFFFFFFFFFFFFFn);
end = start | bitmask;
}
} else {
// IPv4
if (mask === 32) {
end = start;
} else {
const bitmask = (0xFFFFFFFF >>> mask);
start = (start & ~bitmask)>>>0;
end = (start | bitmask)>>>0;
}
}
ranges = [[start, end, mask]];
}
if (ipNum && (ipNum < ranges[0][0] || ipNum > ranges[0][1])) {
return null;
}
ranges = ranges.map(([s, e, m]) => [numToHex(s), numToHex(e), m]);
if (ip) {
ranges = ranges[0];
}
return ranges;
}

63
src/utils/ipMiddleware.js Normal file
View File

@ -0,0 +1,63 @@
/*
* express middleware to add IP related getters to Request object
* req.ip -> ip as string, IPv6 cut to 64bit block
* req.ipHex -> ip as hex string
* req.ipNum -> ip as BigInt if IPv6 and Number if IPv4
* req.cc -> two character country code based on cloudflare header
*/
import { USE_XREALIP } from '../core/config';
import { isIPv6, unpackIPv6, ipToHex } from './ip';
const ipGetter = {
get() {
let ip;
if (USE_XREALIP) {
ip = this.headers['x-real-ip'];
}
if (!ip) {
ip = this.connection.remoteAddress;
if (USE_XREALIP) {
// eslint-disable-next-line no-console
console.warn(
`Connection not going through reverse proxy! IP: ${ip}`,
);
}
}
if (isIPv6(ip)) {
ip = `${unpackIPv6(ip).slice(0, 4).join(':')}::`;
}
this.ip = ip;
return ip;
},
};
const ipHexGetter = {
get() {
return ipToHex(this.ip);
},
};
const ipNumGetter = {
get() {
const ipHex = `0x${this.ipHex}`;
return (ipHex.lengh > 10) ? BigInt(ipHex) : Number(ipHex);
},
};
const ccGetter = {
get() {
if (!USE_XREALIP) {
return 'xx';
}
const cc = this.headers['cf-ipcountry'];
return (cc) ? cc.toLowerCase() : 'xx';
}
}
export default (req, res, next) => {
Object.defineProperty(req, 'ip', ipGetter);
Object.defineProperty(req, 'ipHex', ipHexGetter);
Object.defineProperty(req, 'ipNum', ipNumGetter);
Object.defineProperty(req, 'cc', ccGetter);
next();
};

View File

@ -4,12 +4,12 @@
import net from 'net';
import { isIPv6, ip4InRangeToCIDR } from './ip';
import { isIPv6, ipSubnetToHex } from './ip';
import { OUTGOING_ADDRESS } from '../core/config';
const WHOIS_PORT = 43;
const QUERY_SUFFIX = '\r\n';
const WHOIS_TIMEOUT = 30000;
const WHOIS_TIMEOUT = 10000;
/*
* parse whois return into fields
@ -18,14 +18,13 @@ function parseSimpleWhois(whois) {
let data = {
groups: {},
};
const groups = [{}];
const text = [];
const lines = whois.split('\n');
let lastLabel;
for (let i = 0; i < lines.length; i += 1) {
const line = lines[i].trim();
let line = lines[i].trim();
if (line.startsWith('%') || line.startsWith('#')) {
/*
* detect if an ASN or IP has multiple WHOIS results,
@ -39,6 +38,11 @@ function parseSimpleWhois(whois) {
continue;
}
if (line) {
// strip network prefix for rwhois
if (line.startsWith('network:')) {
line = line.slice(8);
}
const sep = line.indexOf(':');
if (~sep) {
const label = line.slice(0, sep).toLowerCase();
@ -86,25 +90,17 @@ function parseSimpleWhois(whois) {
* @return object with whois data
*/
function parseWhois(ip, whoisReturn) {
const data = {};
if (!whoisReturn) {
return data;
}
const whoisData = parseSimpleWhois(whoisReturn);
let cidr;
if (isIPv6(ip)) {
const range = whoisData.inet6num || whoisData.netrange || whoisData.inetnum
|| whoisData.route || whoisData.cidr;
cidr = range && !range.includes('-') && range;
} else {
const range = whoisData.inetnum || whoisData.netrange
|| whoisData.route || whoisData.cidr;
if (range) {
if (range.includes('/') && !range.includes('-')) {
cidr = range;
} else {
cidr = ip4InRangeToCIDR(ip, range);
}
}
}
let range = whoisData.inetnum || whoisData.inet6num || whoisData.cidr
|| whoisData.netrange || whoisData.route || whoisData.route6
|| whoisData['ip-network'] || whoisData['auth-area'];
range = ipSubnetToHex(range, ip);
if (range) data.range = range;
let org = whoisData['org-name']
|| whoisData.organization
|| whoisData.orgname
@ -117,52 +113,81 @@ function parseWhois(ip, whoisReturn) {
if (contactGroup) {
[org] = whoisData.groups[contactGroup].address.split('\n');
} else {
org = whoisData.owner || whoisData['mnt-by'] || 'N/A';
org = whoisData.owner || whoisData['mnt-by'];
}
}
const descr = whoisData.netname || whoisData.descr || 'N/A';
const asn = whoisData.asn
if (org) data.org = org;
const descr = whoisData.netname || whoisData.descr;
if (descr) data.descr = descr;
let asn = whoisData.asn
|| whoisData.origin
|| whoisData.originas
|| whoisData['aut-num'] || 'N/A';
let country = whoisData.country
|| (whoisData.organisation && whoisData.organisation.Country)
|| 'xx';
if (country.length > 2) {
country = country.slice(0, 2);
|| whoisData['aut-num'];
if (asn) {
// use only first ASN from possible list
asn = asn.split(',')[0].split('\n')[0]
// only number
if (asn.startsWith('AS')) {
asn = asn.slice(2);
}
const dotIndex = asn.indexOf('.');
if (dotIndex === -1) {
// asplain
asn = parseInt(asn, 10);
if (!Number.isNaN(asn)) {
data.asn = asn;
}
} else {
// asdot
const p1 = parseInt(asn.slice(0, dotIndex));
const p2 = parseInt(asn.slice(dotIndex + 1));
if (!Number.isNaN(p1) && !Number.isNaN(p2)) {
data.asn = (p1 << 16) | p2;
}
}
}
return {
ip,
cidr: cidr || 'N/A',
org,
country,
asn,
descr,
};
let country = whoisData.country
|| whoisData.organisation?.Country
|| whoisData['country-code'];
if (country) {
data.country = country.slice(0, 2).toLowerCase();
}
return data;
}
/*
* send a raw whois query to server
* @param query
* @param host
* @param hostInput host with or without port
*/
function singleWhoisQuery(
query,
host,
hostInput,
) {
if (host.endsWith(':4321')) {
throw new Error('no rwhois support');
const options = {
timeout: WHOIS_TIMEOUT,
};
const pos = hostInput.indexOf(':');
if (~pos) {
// split port if neccessary
options.host = hostInput.slice(0, pos);
options.port = hostInput.slice(pos + 1);
} else {
options.host = hostInput;
options.port = WHOIS_PORT;
}
if (OUTGOING_ADDRESS) {
options.localAddress = OUTGOING_ADDRESS;
options.family = isIPv6(OUTGOING_ADDRESS) ? 6 : 4;
}
return new Promise((resolve, reject) => {
let data = '';
const socket = net.createConnection({
host,
port: WHOIS_PORT,
localAddress: OUTGOING_ADDRESS,
family: 4,
timeout: WHOIS_TIMEOUT,
}, () => socket.write(query + QUERY_SUFFIX));
const socket = net.createConnection(
options,
() => socket.write(query + QUERY_SUFFIX),
);
socket.on('data', (chunk) => { data += chunk; });
socket.on('close', () => resolve(data));
socket.on('timeout', () => socket.destroy(new Error('Timeout')));
@ -204,30 +229,33 @@ function checkForReferral(
return null;
}
/*
/**
* whois ip
* @param ip ip as string
* @param host whois host (optional)
* @returns {
* range as [start: hex, end: hex, mask: number],
* org as string,
* descr as string,
* asn as unsigned 32bit integer,
* country as two letter lowercase code,
* referralHost as string,
* referralRange as [start: hex, end: hex, mask: number],
* }
*/
export default async function whoisIp(
ip,
host = null,
) {
let useHost;
if (!host) {
if (Math.random() > 0.5) {
useHost = 'whois.arin.net';
} else {
useHost = 'whois.iana.org';
}
} else {
useHost = host;
}
let whoisResult = '';
export default async function whoisIp(ip, options)
{
let host = options?.host || 'whois.iana.org';
let logger = options?.logger || console;
let whoisResult;
let prevResult;
let prevHost;
let refCnt = 0;
while (refCnt < 5) {
let queryPrefix = '';
if (useHost === 'whois.arin.net') {
if (host === 'whois.arin.net') {
queryPrefix = '+ n';
} else if (useHost === 'whois.ripe.net') {
} else if (host === 'whois.ripe.net') {
/*
* flag to not return personal information, otherwise
* RIPE is gonna rate limit and ban
@ -237,18 +265,37 @@ export default async function whoisIp(
try {
// eslint-disable-next-line no-await-in-loop
whoisResult = await singleWhoisQuery(`${queryPrefix} ${ip}`, useHost);
whoisResult = await singleWhoisQuery(`${queryPrefix} ${ip}`, host);
const ref = checkForReferral(whoisResult);
if (!ref) {
break;
}
useHost = ref;
prevResult = whoisResult;
prevHost = host;
host = ref;
} catch (err) {
// eslint-disable-next-line no-console
console.error(`Error on WHOIS ${ip} ${useHost}: ${err.message}`);
logger.error(`WHOIS ${ip} ${host}: ${err.message}`);
host = prevHost;
break;
}
refCnt += 1;
}
return parseWhois(ip, whoisResult);
let result = parseWhois(ip, whoisResult);
if (!result.range) {
// eslint-disable-next-line no-console
logger.error(`WHOIS ${ip} ${host}: This host gives incomplete results.`);
host = prevHost;
}
if (prevResult) {
const pastWhois = parseWhois(ip, prevResult);
if (host && pastWhois.range) {
result.referralHost = host;
result.referralRange = pastWhois.range;
}
result = {...pastWhois, ...result};
}
result.ip = ip;
return result;
}

View File

@ -1,62 +0,0 @@
/* @flow */
// this script checks a ip with the pixelplanet proxychecker
//
import fetch from '../src/utils/proxiedFetch.js';
import isoFetch from 'isomorphic-fetch';
/*
* check proxycheck.io if IP is proxy
* Use proxiedFetch with random proxies
* @param ip IP to check
* @return true if proxy, false if not
*/
async function getProxyCheck(ip: string): Promise<boolean> {
const url = `http://proxycheck.io/v2/${ip}?risk=1&vpn=1&asn=1`;
//const url = 'http://pixel.space';
console.log('fetching proxycheck', url);
const response = await fetch(url, {
headers: {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36',
}
});
if (!response.ok) {
const text = await response.text();
console.log('proxycheck not ok ' + response.status + '/' + text);
return;
}
const data = await response.json();
console.log('proxycheck.io is proxy?', ip, data);
const ret = data.status == 'ok' && data[ip].proxy === 'yes';
console.log(ret);
}
async function getIPIntel(ip: string): Promise<boolean> {
const email = Math.random().toString(36).substring(8) + "-" + Math.random().toString(36).substring(4) + "@gmail.com";
const url = `http://check.getipintel.net/check.php?ip=${ip}&contact=${email}&flags=m`;
console.log('fetching getipintel', url);
const response = await fetch(url, {
headers: {
Accept: '*/*',
'Accept-Language': 'de,en-US;q=0.7,en;q=0.3',
Referer: 'http://check.getipintel.net/',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36',
}
});
// TODO log response code
if (!response.ok) {
const text = await response.text();
console.log('getipintel not ok ' + response.status + '/' + text);
return;
}
const body = await response.text();
console.log('fetch getipintel is proxy?', ip, body);
// returns tru iff we found 1 in the response and was ok (http code = 200)
const value = parseFloat(body);
return value > 0.995;
}
const ip = '188.172.220.70';
getProxyCheck(ip);
getIPIntel(ip);

View File

@ -1,6 +0,0 @@
email="cxyvsf@gmail.com"
while read i; do
RESULT=`geoiplookup $i | sed -e 's/.*, //'`
#PROXY=`wget "http://check.getipintel.net/check.php?ip=$i&contact=alpha@gmail.com" -qO -`
echo "$i $RESULT"
done < ./ips

View File

@ -1,4 +0,0 @@
email="cxyvsf2@gmail.com"
RESULT=`geoiplookup $1 | sed -e 's/.*, //'`
PROXY=`wget "http://check.getipintel.net/check.php?ip=$1&contact=alpha@gmail.com" -qO -`
echo "$1 $RESULT $PROXY"

Binary file not shown.

After

Width:  |  Height:  |  Size: 22 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 43 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 22 KiB