How do you instruct AWS FARGATE to provision a t2.small via ECS CLI instead of the default micro?

The short answer is "you don't have to".

The long answer is I was doing it wrong. Here's what was wrong and what I learned.

After working through tutorials, RTFMing and re-building my ECS configuration I found that I actually wasn't launching via Fargate. I had made a mistake during the initial configure command, so the default config was using EC2 launch type. When making the cluster it created an Autoscaling Group template with t2.micro in it, and then it was using this to launch the initial instance. I didn't think this was unusual, as I read that Fargate managed your EC2's for you, which I presumed was what was happening. No, this was "traditional" EC2 creation via a scaling template.

I figured out the mistake by viewing the config file. I noticed ecs-cli doesn't have any "view config" commands - they only write to the config, so it feels a bit mysterious when you are issuing later commands, especially as you are encouraged to set lots of defaults (presumably to simplify the CLI commands which get long and cumbersome).

less ~/.ecs/config

version: v1
default: MY-TRIAL-CONFIG
clusters:
  MY-TRIAL-CONFIG:
    cluster: MY-TRIAL-CLUSTER
    region: ap-southeast-2
    default_launch_type: ""      <= this was the problem
  tutorial:
    cluster: tutorial
    region: ap-southeast-2
    default_launch_type: FARGATE

So I then tried the EC2-only parameter:

ecs-cli up --instance-type "t2.small" --capability-iam

and this changed the Autoscaling Group and indeed launched a t2.small. So I now know how to do it the EC2 way.

To do it properly as I had originally intended with Fargate, I needed default-launch-type FARGATE:

ecs-cli configure --cluster MY-TRIAL-CLUSTER --region ap-southeast-2 --default-launch-type FARGATE --config-name MY-TRIAL-CONFIG

Now the config looks like this:

less ~/.ecs/config

version: v1
default: MY-TRIAL-CONFIG
clusters:
  MY-TRIAL-CONFIG:
    cluster: MY-TRIAL-CLUSTER
    region: ap-southeast-2
    default_launch_type: FARGATE
  tutorial:
    cluster: tutorial
    region: ap-southeast-2
    default_launch_type: FARGATE

After doing this, ecs-cli compose service up works very differently from the observed ECS launch flow.

it doesn't create any Autoscaling stuff
it doesn't launch any EC2 instances (I thought Fargate did, but managed it for you, but you don't get to see the container hosts)
It created a Cloudformation Stack instead (interestingly with a t2.micro EcsInstanceType parameter)
It's very clearly labelled on the Cluster home page: there's a row for EC2 launch types below the FARGATE row.

e.g.

MY-TRIAL-CLUSTER >
FARGATE
1          0             1
Services   Running tasks Pending tasks

EC2
0        0             0                                              0
Services Running tasks Pending tasks CPUUtilization MemoryUtilization Container instances

It's very clear now I see it. There are no instances for Fargate services.

You can also watch Fargate re-start the task-containers if they fail, either in the control panel, or with ecs-cli compose service ps and ecs-cli logs --task-id XXX --follow, to get a feel for the magic and how the Docker Compose features are being performed.


How to determine actual cost of specific spot instance

From Amazon EC2 Spot Instances Pricing:

  The following table displays the Spot price for each region and
  instance type (updated every 5 minutes).

So one (crude) option would be to parse that page to get the price matching your particular instance type and region.

I don't know if the equivalent info is available via the AWS Price List API:

the Bulk API clearly doesn't support it:

  Offer files do not include information about expiring free tier offers
  or Amazon EC2 Spot Instances.

the spot instace pricing isn't explicitly declared as unsupported in the Query API, so there might be a chance for another option here, that'd be better than parsing pages. But I didn't try - I'm not an AWS user yet.


How do I create separate versions of lambda functions through terraform?

I just needed to set

publish = true

I then enconter another issue for creating seprat versions but finding this answers my initial question


Installing azcopy in Gitlab Runner (Alpine Linux)

This is a glibc dependency issue on Alpine.

This azcopy binary is compatible with Linux distributions that use glibc, GNU's C standard library. Alpine Linux uses a different libc implementation, musl-libc, which is generally not compatible to glibc. When trying to run non-Alpine-built binaries on Alpine, they'll usually fail to link since the glibc shared object, libc.so.6, is missing. azcopy: not found is actually the result of the dynamic linker failing to resolve the requires libraries.

You could verify by running ldd ./azcopy:

    /lib64/ld-linux-x86-64.so.2 (0x7fa0bc9c3000)
    libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7fa0bc9c3000)
    libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7fa0bc9c3000)

Note the libc.so.6 dependency. On vanilla Alpine, there's only the musl libc binary, libc.musl-x86_64.so.1.

Fortunately, Alpine provides a light compatibility layer for glibc, through the libc6-compat package. It installs a libc.so.6 link which is redirected to the musl shared object, and adds some missing glibc interface functions. This seems to be enough to running azcopy.

Therefore, for running azcopy on Alpine, just install the libc6-compat package, using:

apk add libc6-compat


Docker swarm replica stays 0/1 and service stays in status new

Try do pull your image manually first.
docker pull ${DOCKER_REGISTRY}imageNameX:${VERSION}

And then re-run your compose-file
It may be a bug using a private registry, or something simple as you don't have enough disk space to pull the image.


What's the best way to restrict an environment to a specific CDN?

If the different CDNs are hosted at different IP addresses one possibility might be to configure the local hosts table for the machines in a certain environment in such way as to overwrite the DNS resolution for the CDNs which shouldn't be accessible and pointing them to either:

a non-existent address, causing connection timeout failures if the invalid CDNs are referenced
a server where the respective files cannot be found, causing 404 errors (faster than the connection timeouts)


why use multiple From instructions in multistage build?

First of all, you don't need the COPY instructions or multiple RUN instructions if you have only one stage:
FROM diamol/base AS build-stage
RUN echo 'Building...' > /build.txt && echo 'Testing...' >> /build.txt
CMD cat /build.txt

Of course, the textbook example is just that - an example to illustrate the concept.
In reality multistage builds are used to create smaller final images, while keeping the entire build process inside the container.
Generally speaking one may install all development and runtime dependencies in a build stage, then copy just the final distribution package to the final stage, where only runtime dependencies are installed to reduce the final image size.
For example:
FROM ubuntu:latest AS base
RUN apt -y install some-library

FROM base AS build
RUN apt -y install some-library-dev
COPY . /src
WORKDIR /src
RUN make && make install

FROM base
COPY --from=build /usr/bin/app /usr/bin/app
CMD /usr/bin/app


Docker actually provides DNS support for networking between containers. This means that if you define your database connection as:

http://redis:6379

This will resolve redis to the correct network address, given that you fulfill the following:

All containers are on the same network
The containers have been named

This can either be done manually by starting both instances via the cli or automated with a docker compose script, which is my recommendation. In the compose script, from version 3, you would define the both services (aka the containers):

version: "3"
services:
  web:
    build: ./web
    container_name: web
    networks:
      - backend
    command: bash -c "./start.sh"

  redis:
    image: redis:3.2
    container_name: redis
    networks:
      - backend

And at the end of the file, define the network both instances should be connected to:

networks:
  backend:


problems running Ansible with become on domain-connected Windows Server

My problem resolves down to an issue with group policy, and is likely to be solved with the same answer as this question:

https://serverfault.com/questions/695839/error-1385-when-executing-runas-on-windows-enterprise-7

i.e. the required permissions for become to work on Windows, seem to be:

Allow Logon as a Service
Allow Logon Locally

Alternate Solution

reference

  batch: Runs the process under a batch context that is similar to a
  scheduled task with a password set. This should bypass most WinRM
  restrictions and is useful if the become_user is not allowed to log on
  interactively.

There is another permission called 'Allow Logon as a Batch Job". If the domain-user has this permission, the user can run scripts as though they were scheduled tasks.

You can run your scripts using Ansible become and change the logon type to batch as shown below:

win_whoami:
    become: yes
    become_user: ''SOMEDOMAIN\SOMEUSER''
    become_method: runas
    become_flags: logon_type=batch


Troubleshooting VPC flow logs with an S3 bucket using SSE-KMS encryption with CMK

Turns out I was missing one very important line in my KMS key policy:

    resources = ["*"]

Now it works fine, and my full policy looks like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Allow VPC Flow Logs to use the key",
            "Effect": "Allow",
            "Principal": {
                "Service": "delivery.logs.amazonaws.com"
            },
            "Action": [
                "kms:ReEncrypt",
                "kms:GenerateDataKey",
                "kms:Encrypt",
                "kms:DescribeKey",
                "kms:Decrypt"
            ],
            "Resource": "*"
        }
    ]
}


Is it a common practice to include the version of an API into the rest interface itself?

Many APIs include a version number in the url of the service. Some examples:

https://api.twitter.com/1.1/statuses/home_timeline.json
https://graph.facebook.com/v2.2/me/adaccounts
https://vision.googleapis.com/v1/images:annotate?key=YOUR_API_KEY
https://maps.googleapis.com/maps/api/js?v=3.29
https://westeurope.api.cognitive.microsoft.com/vision/v1.0/analyze
https://www.europeana.eu/api/v2/opensearch.rss?searchTerms=Rembrandt

However many people on the Internet argue that this is incorrect because it violates REST/HTTP specifications; "embedding of API version into the URI would disrupt the concept of hypermedia as the engine of application state". It’s more appropriate to use the Accept header to request a particular version. For the latter people suggest to use a Vendor MIME type, for example something like Accept: application/vnd.mycompany-api-v3+json

Nevertheless, the reason why many API designers do include an API version in the url is because it's easy to implement and debug (clearly visible which version you are using).

Side note: in our organization we recently decided to support both a versionless endpoint that always point to the latest stable version of our APIs, as well as an endpoint with a version specification. This allows users to choose if they want to commit to our latest API (which may require making changes when a newer version is released), or to connect to a fixed API version (which may become deprecated over time).


trigger jenkins jobs only when a specific branch is updated

Install the git plugin, add credentials for the account you want it to use, then set the job to use SCM Polling. It will check on a preset interval for changes to that repo and trigger when changes are detected. If you need instant triggering upon push, you're looking for webhooks, which is significantly more difficult to set up. I find that polling every 2 minutes or so is good enough in most cases. Here's a decent example: https://www.baeldung.com/jenkins-job-schedule


Update 12/05/2020: Fargate prices have been greatly reduced over the last months and are now comparable with EC2 prices for the same CPU / RAM configurations. That makes Fargate a valid option for 24x7 workloads.

Originally written in 2018 - no longer correct!

Fargate is more expensive than EC2 for the same vCPU/RAM amount.

For example:

m5.large (2 vCPU, 8 GB RAM) costs $0.096/hr which is ca $69.12/month
Fargate container with 2vCPU and 8 GB RAM costs $0.2028/hr or $146.02/month

If you want to run your container 24x7 you'll be much better off running it on an EC2 instance.

However if your containers only run briefly to complete a task and then exit, or if they scale up and down based on demand it will be much easier for you to run them in Fargate - you won't need to scale up and down the underlying EC2 cluster to support the load.

With Fargate you pay premium for the flexibility.

In many cases it works out better to run on Fargate even if it's more expensive - we spin up batches of hundreds of containers at a time a couple times per day for some processing and each container runs for only about 10 minutes. If we had to scale up the EC2/ECS cluster before each run, wait for it to settle, deal with the failures, then run our batch job and then scale down again the overhead would be quite high and our batch processing would take much longer.

Here Fargate works great for us. I wouldn't use it for an always-on service though.

Hope that helps :)


Using Jenkins declarative when conditions in scripted pipelines

It would seem not - in the scripted pipeline, groovy needs to be used - see the docs

node {
    stage('Example') {
        if (env.BRANCH_NAME == 'master') {
            echo 'I only execute on the master branch'
        } else {
            echo 'I execute elsewhere'
        }
    }
}

(taken straight from the docs)

This would be the same as when { branch 'master' }


how to automatically create a subdomain for every pull request

We do this at work.

We have a small server, let's call it the receiver, it's the target of the GitHub webhook events. It runs a small application that parses the payload and incorporates logic around how to proceed e.g. create a new server on the infrastructure provider, update the load balancer, deploy to an existing server, destroy the server etc. This could be a traditional web application serving up an API or it could be a serverless application, up to you how you want to approach it.

The receiver is relatively straightforward to handle, the other necessary supporting systems you'll need are an approach to configuration management/provisioning (how does the server have the packages necessary to run the application), secrets management (how does the server get access to sensitive information) and routing  (how do the subdomains get updated and route to the correct server).

It could be worthwhile looking at preparing an AMI with the necessary services configured, a CloudFormation template with the infrastructure provisioning logic and CodeDeploy could handle deployments for you.

Configuration Management

This is really up to you and your team, there are a multitude of tools you can use or you can simply rely on shell scripting. At what point in the servers lifecycle you apply the changes is what's discussed in the AMI design article I linked.

Secrets Management

This is a challenging topic to address with the information at hand, in the interest of brevity I'll leave that up to you and your team.

Routing

There are a few ways you can handle routing, the Application Load Balancer (not to be confused with the ELB/NLB) offered by AWS supports host based routing. Alternatively you could use a reverse proxy like NGINX or HAProxy, when you provision a new environment you will need to update this routing (ideally automatically) irrespective of what approach you take.

Don't forget to consider how you'll handle the database/persistence layer & zero down time deployments. The question to ask with the persistence layer is whether the team will share a database and how that will interact with things like migrations. On the topic of zero downtime deployments, CodeDeploy should handle that nicely for you. One more thing, you mentioned a single mobile application pointing to different environments, how will you point these applications to the environments.


How to use remote Ivy artefacts in Maven jobs with different binary repository managers?

I'd go with (2). Since Bob has a more rigid setup (requires Maven metadata), and Alice can provide that (using more flexible Ant), that sounds like the easiest way to go.


Monorepos have been getting a lot of attention in recent years since google popularized them.  If you aren't on a similar scale as google it might not work out as well for you.  I've been in a startup that moved to monorepos and we quickly found that:

most tools are built with one repo leading to one artifact that gets deployed.  You will have to rework all of those tools to handle the monorepo.  There are tools that are designed for monorepos.
a monorepo puts all of your eggs in one basket.  If somebody steals a laptop would you rather them get half of your code or all of it?  Keeping different things in their own repos reduces exposure.
your CI for code validation is now much more complicated.  You need to support multiple languages in your code conformance tests.  Developers will get upset when you check backend stuff and they've only made frontend changes.  Now you're stuck trying to figure out how to disable some CI checks, but make sure they're enabled when needed.

All of this adds up to a load of work for release engineers.  Do you have people focused on release engineering with free time to tackle multiple complicated projects?  Google can make this work because they have hordes of engineers and they can dedicate ten folks to this sort of thing for months until it is smooth enough to develop on top of.  If you are not at the same scale as google you might not be able to follow their example.  The folks that have succeeded with this tend to be big organizations.  You may not have found an open source example because none exists.
These points are also noted in this article:

The frank reality is that, at scale, how well an organization does with code sharing, collaboration, tight coupling, etc. is a direct result of engineering culture and leadership, and has nothing to do with whether a monorepo or a polyrepo is used. The two solutions end up looking identical to the developer. In the face of this, why use a monorepo in the first place? Please don’t!

So focus on improving your engineering culture and don't get worried about not following the latest tech trends.


Build Docker Images with Ansible + Vagrant

If it's within scope, I would use Packer to build the artifacts.

Write a Packer template using the builders that you want:

Vagrant builder for the Vagrant image
Docker builder for the Docker image.

Use the same Ansible playbook as you already have to provision each of the builders.

Note that you need to make sure that the Ansible playbook will work in both types of virtualised environments.

This has the benefit of producing arbitrary types of artifacts, based off of a single input. You can use a single workflow for both the Vagrant and the Docker images, and they will be built off of the same state of the codebase (i.e. git hash).


Error when creating new recource using backend AWS s3

Guys I discovered what happened.
It was necessary to apply these two resources that I informed (project_data and environment_data), but they were in the above directories and I was not aware.

Something like this:

Application
|
|-Environment
  |-environment_data.tf
|
|- project_data.tf


One way to solve this it to use multiple pipelines. The first pipeline runs the test in docker. When that has been completed, one could call another job

https://jenkins.io/doc/pipeline/steps/pipeline-build-step/

https://support.cloudbees.com/hc/en-us/articles/221400287-How-to-pass-parameter-to-downstream-job-in-Pipeline-job-

node {
    paramAValue = "paramAValue"
    paramBValue = "paramBValue"
    build job: 'downstream-freestyle', parameters: [[$class:  'StringParameterValue', name: 'ParamA', value: paramAValue], [$class:
       'StringParameterValue', name: 'ParamB', value: paramBValue]]
}

that runs docker-compose on Jenkins itself.


Measure service unavailability during upgrade

What you suggest might be the simplest way to collect the data, but you will have to do quite a bit of work to extract the availability over certain periods.

I think it's fair to say that if you want availability, you need a monitoring system. This means having an extra service in your catalogue to continuously probe the availability of your microservices over time. Storing them in a time-series database would allow you to make queries to establish availability over various periods.

There are many tools which could do this for you. A good starting place would be the CNCF monitoring landscape


Same Docker image, multiple configurations

One could pass environment variables when running docker, e.g.:

  Additionally, the operator can set any environment variable in the
  container by using one or more -e flags, even overriding those
  mentioned above, or already defined by the developer with a Dockerfile
  ENV:

$ docker run -e "deep=purple" --rm ubuntu /bin/bash -c export


How to handle different credentials in continuous integration?

I'm not familiar with teamcity, but you can :

add a config folder in your repo, and inside this repo
create three files (staging.yml, preprod.yml, prod.yml).
on each file you will write down variables specific to each
environment, and apply them within pipeline.

For secrets, it's cautious to handle that manually or better use Vault.

To not throw on error your pipeline/pods, set secrets as optional that could help if you want to set them later.


How to find out the used Key pair name via AWS CLI?

For a given instance, you would first use aws ec2 describe-instances to get the information JSON for your instance.

The information also contains the keypair name used to create that instance.

E.g. for an instance i-0e2x8xd7xxx  (Note: I use the awesome tool jq to do JSON parsing but you can use any other solution)

aws ec2 describe-instances --instance-ids i-0e2x8xd7xxx | jq '.Reservations[].Instances[].KeyName'

Output:

"my_key_name"

You can store that in a variable, say $keypair_name and then pass it into your aws ec2 get-password-data command.

You would also need to pass in the path on your machine where your keypairs are located e.g. $keypair_path.

For example:

aws ec2 get-password-data --priv-launch-key $keypair_path/$keypair_name .....


Cannot connect to server running on host from Dockerfile / image build

First you have to start netcat listening in the background.
$ netcat -l 3440 &

Then build your Dockerfile.

#
# Ubuntu Dockerfile
#
# https://github.com/dockerfile/ubuntu
#

# Pull base image.
FROM ubuntu:14.04

# Install.
RUN \
  sed -i 's/# \(.*multiverse$\)/\1/g' /etc/apt/sources.list && \
  apt-get update && \
  apt-get -y upgrade && \
  apt-get install -y build-essential && \
  apt-get install -y software-properties-common && \
  apt-get install -y byobu curl git htop man unzip vim wget && \
  rm -rf /var/lib/apt/lists/*

# Set environment variables.
ENV HOME /root

# Define working directory.
WORKDIR /root
RUN apt-get install -y netcat
RUN nc localhost 3440 | tar -x -O > ores-$(date +%s.%N).tgz
# Define default command.
CMD ["bash"]


How to make Jenkinsfile credential visible to many stages

No, the credentials will only be visible within the block passed to withCredentials, not outside of that.
There is no way to make the credentials globally available without switching to Scripted Pipeline.  With Scripted, you can wrap your entire job in withCredentials.  This is not possible with Declarative.


Why do companies use JFrog Artifactory or other repositories instead of Maven central repository?

Saved bandwith and faster downloads: Artifactory stores the artifacts that are downloaded from maven central. So if another developer needs the same dependencies they don't need to be downloaded again from maven central but instead they can be delivered from the local artifactory instance.
This makes downloading faster because company networks are usualy faster then the internet connection and it also saves bandwith.
As example imagine that a developer adds a new dependency to a project. This dependency then needs to be downloaded into the local maven cache of every other developer that is working on the same project.
On large projects with a lot of developers this means that the dependency might be downloaded 30 times.
Stable builds: Dependencys might be removed from maven central. If you have them in your local artifactory instance the project who depends on it can still be build.
Easier maintenance: Another benefit that has nothing to do with maven: Artifactory provides several types of repositories like for Docker or Vagrant.
So instead of setting up a maven artifacts repository, a docker registry and some kind of vagrant store you could create three repositories in artifactory.
Thus you only need to maintain one application (artifactory) instead of three separate applications.

Some or all of these benefits might also apply to other binary repository managers but as I only know artifactory I can't say something about that.


How to define Service Level Objectives without historical data?

Without much data you are really guessing, the general thought I have heard before is to start at 80% SLO then work your way up as you have data to back up your goal. There really is not set way of figuring out a SLO without some data. Even taking and using a week's data would be good enough to give you a reasonable idea.


How to avoid continuous integration-caused instabilities in test environments?

I'll give my experience on this one, mostly because it showcases why some answers are not always applicable.

Some context to start:

We have 7 environments to host roughly 80 applications, most of them rely on each others through webservices or shared tables on db2-iSeries.
For good or bad, the iSeries are our DB system of reference.
This last point invalidates any idea of bringing the app with its dependencies in an isolated environment as bringing up an AS400 for each would cost too much and we wouldn't have the hardware to run it anyway.

What we are doing is not a complete automated Continuous Delivery, we have a schedule of releases to bring up a coherent lot of applications for the general operations. Aside from this each test teams can trigger a release in one of the Q/A environment for the application they are testing and can put a lock on some application version to avoid another team request breaking their tests.

Dependencies of applications are checked before release, so the system won't release something if other applications can't be updated or doesn't match its dependencies needed.
The main idea is to allow updates when it won't impact someone, if there are no tests planned, it should flow from previous environment (and we're aiming at removing the scheduled releases in the 5 firsts environments on mid term now we have validated this 'on demand' method system).

The short version is to have a 'semaphore' system around the applications in the environment, a team should be able to lock its target application with its dependencies (and transitive dependencies) for the time of manual tests.
The implementation of this semaphore is highly dependent on your automation system so I won't extend on that.

Of course the easy way is, as others mentioned, to create a fresh environment for an application with all its dependencies to avoid the semaphore described above.


Jenkins & Bitbucket; cancel previous build if new commit made to same branch?

Note: this answer comes from my background of building custom solutions, it's not a config-only one, which, if available, would obviously be preferable. But it's maybe something to consider otherwise.

You could teach the script executing the unit test job to persist a per-branch execution state.

When the script is launched it would need to obtain the branch name from its invocation parameters. It would then obtain the persisted state for the branch.

If the state is either missing or not running it would set it to running + job ID, execute the unit test and then set the state to not running (or delete it).

If the state is running + job ID it means another unit test is already running and you have the corresponding job ID. The script can then (gracefully?) terminate the already running job by its ID and take it's place, updating the state to running + new job ID.

But IMHO terminating a job in progress (which may be very close to completion) is not ideal - it leaves room for constantly killing jobs and launching news ones only to be, at their own turn, killed by subsequent jobs - all without actually completing any of them.

Instead I'd modify the logic to:

leave the job in progress alone and instead just record in the persisted state a pending job ID (either a unique value, overwritten to reflect the most recent pending job submitted or a list of all pending jobs, sorted by submission time)
when the job in progress completes it checks the pending job ID values and, if any, launches the most recent one (or the oldest one, or even follow some other selection logic, if using a list)

This approach avoids wasting unit test resources and can easily be adjusted even for more than one but still limited resources.


How to install aws cli on a self-hosted azure DevOps Windows agent?

The best way to have AWS CLI tooling installed to your custom agent is by installing from Visual Studio Marketplace. AWS CLI is part of AWS Tools for Azure DevOps Service and Azure DevOps Server (the on-premise Azure DevOps) under the name of "AWS Tools for Microsoft Visual Studio Team Services".

Installing this AWS Tools will enable AWS tooling on CI (build pipeline) and CD (release pipeline).

This is the official landing page of AWS Tools in Visual Studio Marketplace:
https://marketplace.visualstudio.com/items?itemName=AmazonWebServices.aws-vsts-tools

Official documentation is available at: https://docs.aws.amazon.com/vsts/index.html


How do I bootstrap an alpine image to work with a private repo proxy behind TLS?

It turns out that /etc/ssl/cert.pem is the bundle scanned by apk to verify a repo behind HTTPS, so the following works:

wget our.cert.server.com/CA.crt
cat CA.crt >> /etc/ssl/cert.pem
sed -i 's,http://dl-cdn.alpinelinux.org/alpine,https://nexus-endpoint.company.com/repository/alpine-proxy,' /etc/apk/repositories
apk update


What is the advantage of using Prometheus over ingesting metrics using logs?

Efficiency

If you don't need the file it saves space and trouble to leave it out.  Logs require rotation and retention policies which are nice to skip.

Reliability

Skipping the file avoids issues with the file that could caused lost metrics like a full disk.

Latency

Skipping the file means that you remove latency from disk I/O.  For most metrics you don't care if they show up a few seconds late, but if you're trying to build things to respond to some metric quickly.  You might want to put it on Kafka also.


The onboarding experience should be almost as simple as telling your new developer to just clone the repo and run docker-compose up. Personally I wouldn't bother worrying about IDE integration because people might prefer to use different IDEs.

Every project/application (if you have multiple) should be able to run separately and each project/application should have it's own codebase. That's a concept from The Twelve-Factor App methodology.

In the root of each project add a Dockerfile and a docker-compose.yml file.

  NOTE: I created Craft Kits which you can use to automatically generate these files for the framework/language you're using with one command (e.g. mason craft nodejs), you don't even need to know Docker.

The Dockerfile will build your application into an image, installing and dependencies your application needs to run. This image is the image that runs your app.

FROM node:8.8-alpine

# Set our workdir
WORKDIR /app

# Install and cache app dependencies
COPY package.json /app
RUN npm install

# Add project files
COPY . /app

# start app
CMD [ "npm", "start" ]

The docker-compose.yml file will spin up your environment. It will define all the services Docker should start for your application

version: '2'
services:
  web:
    build: .
    volumes:
      - ./:/app
    ports:
      - "3000:3000"

  mongodb:
    image: mongodb
    volumes:
      - ./storage/data/mongodb:/data/db
    ports:
      - "27017:27017"

When you onboard a new developer, you can just tell them to git clone your repo and to fire up their development environment by running:

docker-compose up -d

The advantage of this approach when you are developing locally, running docker-compose up will mount your code as a volume.

Then when you want to deploy, when you build your Dockerfile it will copy your code into your image so everything is packaged up nicely and easily moved around.

We use the same approach for apps on Codemason, my service for building, managing and scaling container-based applications. When you push new code, we automatically build an image from your Dockerfile. Since that image contains everything your app needs to run, it can be easily launched on any server and scaled up/down across multiple servers.


how to run an Ansible command for specific number of hosts

Use the --limit option to select a subset of hosts:

ansible-playbook -i /path  test.yml --tags automation --limit host2

Its description from ansible-playbook documentation:

  further limit selected hosts to an additional pattern


Nexus2: How can I get a list of all artifacts from a specific repository via REST API?

Try this:

# Get status
curl http://localhost:8081/nexus/service/local/status
# List of repositories
curl http://localhost:8081/nexus/service/local/repositories

For the json parameter:

curl -i -H "Accept: application/json" <url>

More information, see this link


ERROR: Unable to read database state: No such file or directory

Please try splitting this line:

RUN apk add git && git clone https://github.com/s0md3v/Photon.git Photon

into two:

RUN apk add git

RUN git clone https://github.com/s0md3v/Photon.git Photon

This will tell you exactly which piece of the command fails. If it is the apk add git part maybe try this: RUN apk update && apk add git.

Also you may fixing to a specific tag in the FROM python:3-alpine as 3-alpine follows the latest alpine release which may have some unknown/unfixed issues.

See https://hub.docker.com/_/python/ for available tags.


What is the major difference between continuous delivery and continuous deployment

"Continous delivery" is the general principle of a development process (including development, testing, etc.) producing software in a way where you could deploy often, ideally at any time. In contrast to a classical waterfall model where you would have fixed deployment dates which would be months or even years apart and planned with huge feature sets. These days, this means that there is some CI/CD pipeline which may or may not include the final step of "deployment".

"Continuous deployment" is a specific aspect which means that the final step of the process, the actual deployment, is automated as well, and happens very frequently, ideally on every commit (after running through a rigorous CI/CD pipeline which may or may not include any human interaction).

There is no versus here. Continuous Deployment is an optional sub-part of Continous Delivery.


You can set up a proxy. A reverse proxy can be a good choice for this kind of setup. You would configure each machine to use the proxy as its apt/pip repository. The first time a package is requested the proxy downloads it from the internet and stores it, the next machine that requires it will get it from the proxy directly.

One popular choice is Squid. Another one is Nginx, set up in a reverse proxy configuration.

The advantage over mirroring entire repositories is that you'll only get the packages you need. The advantage over having Ansible copy the files is that it acts like the real proxy, dependencies and upgrades and all. But you'll  have to change apt and pip repository configurations. Or "hijack" those repository names through your local DNS.


Yes, it's possible. Gitlab has the concept of Discussions (either single comments or discussion threads), which you can have on a variety of development items/activities, including the code itself (in which case they effectively are code reviews):

  The ability to contribute conversationally is offered throughout
  GitLab.

  You can leave a comment in the following places:

  issues
  epics
  merge requests
  snippets
  commits
  commit diffs

But note that only merge request discussions are presently considered resolvable.

Even though the check for a merge request can be configured to be automated, the starting of a review/discussion itself (if/when needed) is a manual operation. From Threaded discussions:

  To start a threaded discussion, click on the Comment button toggle
  dropdown, select Start discussion and click Start discussion
  when you’re ready to post the comment.


PRs not automatically getting trigger in Jenkins github organization

It sounds like maybe you need to configure a webhook for git if you are using the git plugin, but I think that may just build the master branch. I haven't used git much, so you might be able to use the web hook to have it build the pull request - I don't know.

Instead, you might want to check out the pull request builder plugin and see if that does what you are looking for.


How to enforce policies for manifests in Kubernetes?

I would be inclined to implement the Open Policy Agent as this will give you the ability to define a policy similar to this:

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sContainerLimits
metadata:
  name: container-must-have-limits
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
  parameters:
    cpu: "200m"
    memory: "1Gi"

OPA will enforce any policy you defined via an Admission Controller, beware though it's very powerful and I have bricked clusters by not thinking through the options carefully.


Get changes of the parameterized pipeline on the fly in Jenkins

Currently what you are asking for is not possible. This is due to the way that the Jenkinsfile is loaded into the pipeline.
I think dolphy said it best on this StackOverflow answer (I saved this link a while ago to follow it for changes, because I had the same questions):

Jenkins does not know about the new parameters until it retrieves, parses, and runs the Jenkinsfile, and the only way to do that is to...run a build.
In effect, the build history will always be "one run behind" the Jenkinsfile; when you change something in the Jenkinsfile, the next build will run with the "old" Jenkinsfile, but pick up and process the new Jenkinsfile for the build after that.

There is also a related Jenkins Ticket you can follow


How to safeguard Ansible deployment to mitigate accidents?

We're using jobs in jenkins to trigger deployments. It ensures that no matter who does the deployment, the ansible command that is run will be the same. A nice bonus is the build logs record when deployments were triggered, who triggered them and what exactly happened during the deployment.

It's certainly not foolproof, but it's been a nice improvement over running ansible playbooks by hand.

For larger/riskier changes this should ideally be combined with some form of change management so changes are made only after another person/team reviews the change and the approach to the change to help identify and resolve potential issues early.

Additionally it never hurts to have a teammate who understands the change you're making be present and watching while you make big changes so they can watch for and help prevent mistakes in the execution of the change.


Should development tools be installed on the host or in an environment standardized at project level?

At many companies (eg banks) the provenance of the code that gets out to live needs to be known and traceable. This is typically done by having a controlled build farm that builds from source. Release builds upload the binary artefacts to a location where they cannot be overwritten. The artefacts are then deployed by any number of mechanisms that are traceable to an individual and an approved change request. This model is generally just referred to as a ”pipeline”.

Under such a model a ”dev host” or ”dev environment” can have anything and everything that developers want on it. It can use newer versions of software than are available in production. It's entirely down to the context whether that is going to cause problems with code failing testing in a live-like environment. It's typical to put release build code through at least one test environment before pushing it to live. That environment is kept as similar to the live environment as possible.


Whether you allow remote sudo or remote access to something that does SUID root you have a pretty similar attack surface.  I would keep sudo in the chain because it lets you limit the commands easily and has logging that will be vital if you need to audit things later.  sudo also has a much longer history in production.  Doing something else will have less history and higher changes of unpleasant surprises.

There are other things you can do to make this more secure though:

tighten up ssh
only allow the restart commands for a few specific users including one for jenkins
only allow logins for those specific users from all internal IP's or just jenkins and jump box IP's
store logs on remote boxes so they can't be mucked with


Solution that worked for me in the past: create a new image out of the existing one and let entrypoint script of that new image to modify permissions of your volume_mountpoint.

So in the case of your image (php:7.4-apache) you create entrypoint.sh file, something like

chown www-data:www-data -R /path/to/volume_mountpoint
apache2-foreground

And then your Dockerfile should be something like

FROM php:7.4-apache
COPY entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

This way it should correct the permissions when container starts. Note, that I haven't tested the above, might require some tweaks - but hopefully it shows the idea.

P.s. Maybe somebody has a better way of doing this - I would be very interested myself.


Take automatic screenshots of a website and send it to WhatsApp or Facebook Messenger

In case the requirement for Whatsapp/Facebook is not mandatory, you could consider using a solution offering an API for such things and targeted specifically for IT teams.

For example, Slack channels allow uploading files through Slack's API and sharing files in channels.

Documentation: https://api.slack.com/methods/files.upload

For a ready CLI stub, consider for example slackcat.


How can I calculate costs of deploying 1000 instances of Drupal?

In reply to 'Would Amazon EC2 / Google / Azure be cheaper than having dedicated servers?'

I've done a lot of investigating into this and in every case they are NOT.

80 Linux servers running 24h/365 - annual cost:
        Total cost per year
Google  £37,515.46
AWS     £29,196.53

These costs do NOT include data transfer.

You can buy a dedicated cloud appliance for less than £50k, fully supported, that will run more servers and last you 3 years (min).


Accessing a AWS docker instance via localhost only?

Have you tried using awsvpc mode? https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-networking.html
You can follow this tutorial - https://docs.aws.amazon.com/AmazonECS/latest/userguide/create-public-private-vpc.html
You need a VPC, with at least two subnets, one private and one public. After that the sky is the limit ...
If you need internet access from the private subnet, create a NAT Gateway in the public subnet, and route traffic from the private subnet to 0.0.0.0, via the NAT Gateway - https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Scenario2.html
P.S - I never use network mode host with ECS, you should read more about it here - https://docs.docker.com/network/host/ I didn't find a strong use case for using it, except for testing purposes


How to solve `no live upstreams while connecting to upstream` when trying to rproxy jenkins using jwilder/nginx?

After a spending a lot of time I decided not to continue with this approach and decided to use Jenkins helm.


Minecraft docker-container refuses connection

I’m glad my comment helped. I read the article OP mentioned in his answer and reread his question and I think the issue isn’t using -p instead of —publish as the two work the same, but specifying --network host. In Linux’s version of Docker daemon this prevents port publishing from working and since both Docker for Windows and Docker Toolbox use some sort of virtualization to run that version of Docker for Linux containers, I can only assume it has the same effect. The difference being that host in question is not Windows, but whatever Linux is running on top, so the container is completely unreachable.


Thanks for your response @Zeitounator. I'm running Ansible 2.8
Since posting this I have received an answer that works for me. I realized that these facts are presented as a list of dicts as noted in the doco, but I was still having trouble for some reason (confused myself badly.) The solution hinges on using with_items instead of with_dict and using an appropriate conditional like so:

  - name: Test vmware_vm_facts
    vmware_vm_facts:
      hostname: "{{ vcenter_hostname }}"
      username: "{{ vcenter_username }}"
      password: "{{ vcenter_password }}"
      validate_certs: no
    delegate_to: localhost
    register: rc

  - debug:
      msg: "IP of {{ item.guest_name }} ({{ item.uuid }}) is {{ item.ip_address }} and power is {{ item.power_state }}"
    with_items: "{{ rc.virtual_machines }}"
    when: item.guest_name is search(inventory_hostname)


how to start a docker container which dies immediately after startup?

The proper way to run this image, according to the README, is:
docker run -d \
    --name basexhttp \
    --publish 1984:1984 \
    --publish 8984:8984 \
    --volume "$HOME/basex/data":/srv/basex/data \
    --volume "$HOME/basex/repo":/srv/basex/repo \
    --volume "$HOME/basex/webapp":/srv/basex/webapp \
    basex/basexhttp:latest

No weird linking to itself is necessary.
If you wan to run a shell in an already running container (started as above) use:
docker exec -it basexhttp sh

or you can try bash
docker exec -it basexhttp bash

but the container is Alpine-based, so bash might not be installed.
If you want to run a new container and execute shell inside run
docker run -it --rm basexhttp sh
# or bash if it's installed
docker run -it --rm basexhttp bash

The above will remove the container once you exit bash (but the 3 volumes it creates
will remain, you need to find and remove those by yourself).
Edit:
It seems README is wrong about how to start the container. See my answer to this question for more details.


I'm new to Azure DevOps so I may be wrong - I think you are asking about using a Jenkins plugin to scan Azure Repos for Jenkinsfiles.

If you were moving to Azure DevOps, you can store your Azure Pipeline configuration, azure-pipelines.yml,  in YAML format and this can be stored in the code repository (like a Jenkinsfile).

For further pipeline details see; Create your first pipeline and YAML schema reference.

It looks like you can configure Azure Pipelines to scan your organization repos in Azure Repos and GitHub

The GitHub documentation does warn that this feature is actively being improved - so it may change in the future.

For GitHub you need also configure Azure Pipelines access using the GitHub app:

https://github.com/apps/azure-pipelines/installations/new/permissions


Are there any guidelines for when to use gitlfs and will it break the CI?

When to decide whether a file should go to git LFS

Your files can go to git-lfs at any time, but it would be inneficient to implement this solution for few files. Specially if it's basic gradlew jars.

I have used the gradlew strategy previously. The idea is to push this "wrapper" along your code to make the build process portable. Any platform with the proper JRE will be able to build the application in a similar fashion. This removes the headache of enforcing teams to install specific versions of gradle.

The purpose of gi-lfs is to leverage the benefits of git along with the ability to store large files remotely by caching them and indexing them. It reduces the burden of downloading a big repository when binaries are stored.

In my opinion you should start consider this solution when:

Your git repository contains large files (anything in the MB magnitude)
They represent the main content of your repository
These files are updated regularly
It is taking your CI tool minutes to pull/clone the repository

If you consider this solution to track binary versions, I would suggest you go for binary repositories instead such as:

Jfrog Artifactory
Archiva
Nexus

Does the existing CI configuration will break

If you decide to go git-lfs, the normal git commands won't work. You will need to either create symbolic links toward git-lfs after doing the setup or add lfs after git in your CI tool. i.e. git clone becomes git lfs clone.


How to refresh user's content of the site after release?

Here is what works for us.

We have a "version" file deployed with the application itself. Then, application issues a "check version" request periodically that checks if the version that the application has initially loaded is different from the version specified in this version file. If it is different, we show a notification that there is a new version available and suggest to refresh the browser window to get it.

I think that's what Google Inbox, for example, does. They have this periodic checkJsVersion request and, if there is a newer version of the web app available, they show this information box on the bottom left with a "New version is available, hit Refresh to get it" button.

There is also a different approach when the information about a new version of the app is pushed from the server to the client.


What is the purpose of AssumeRolePolicyDocument in IAM?

The purpose of the AssumeRolePolicyDocument is to contain the trust relationship policy that grants an entity permission to assume the role.
In your example it's granting the Lambda service the ability to assume
References
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-iam-role.html
https://docs.aws.amazon.com/cli/latest/reference/iam/create-role.html


Autoscaling on GoCD agents without terminating active builds?

Auto Scaling groups have a useful feature for this, named lifecycle hooks.

Worflow taken from the documentation above:

As you can notice there's a Scale in step, triggering a Terminating:Wait for the autoscaling group and notifying the instance to be terminated, the instance has now to do it's work and once done signal it is ok to be terminated.

If the task can take longer than the autoscaling group HeartbeatTimeout parameter you can reset the timeout with (quoting still from the same page):

  Restart the timeout period by recording a heartbeat, using the
  record-lifecycle-action-heartbeat command or the
  RecordLifecycleActionHeartbeat operation. This increments the
  heartbeat timeout by the timeout value specified when you created the
  lifecycle hook. For example, if the timeout value is 1 hour, and you
  call this command after 30 minutes, the instance remains in a wait
  state for an additional hour, or a total of 90 minutes.

So for your case, the lifecycle notification should startup a kind of script/program which will:

prevent this builder to receive new builds
loop periodically to check if the ongoing builds are done
if there's build still in progress and the timeout is near, reset the timer
if there's no more builds in progress, signal to proceed to termination


How to find future (planned) versions of Docker?

I'll assume you're talking about Docker Engine for Linux.
Docker doesn't plan releases like that currently. What they do is:
Release major and minor versions on GitHub: https://github.com/moby/moby/releases
Put release notes in their docs: https://docs.docker.com/engine/release-notes
The version number is date based on the month they begin the release process, so the current version "20.10" started its release process in October, 2020 and was eventually released in December, 2020. The last major release before that was 19.03.
Finally, you can see what they are working on (features) and what's next on their Roadmap https://github.com/docker/roadmap/projects/1


Googling "system thinking" returned a lot of resources that are about "Systems thinking" rather than "System thinking". This source indicates that "Systems thinking" is one of the three ways that could be applied in DevOps.

After reading several references "Systems thinking" is about interactions rather than silos. If one translates this to DevOps then "Systems thinking" is aligned with the theory and also important for DevOps as this is the interaction between QA, Dev and Ops.

In order to apply "Systems thinking" one should be able to communicate with people from Dev, Ops and QA, listen to them, understand their problems and find ways to let them work together instead of getting a "through it over the wall" mentality and unconnected departments in the company.

An element that really requires "Systems thinking" is Continuous Delivery (CD). Multiple departments are involved, while CI is mostly used by Dev, QA and Ops will be definitely need to be involved in this process as one does not want to release a newer version if integration or manual tests failed and Ops want to be informed if there are changes to the system as they monitor systems and try to prevent and solve issues on the platform.

In summary, "Systems thinking" is required for DevOps engineers. If one only sees Dev, Ops and QA as individual parts, but no interdependency then this is not aligned with the DevOps theory, i.e. intersection between Dev, Ops and QA.

References

https://medium.com/disruptive-design/tools-for-systems-thinkers-the-6-fundamental-concepts-of-systems-thinking-379cdac3dc6a

http://www.thwink.org/sustain/glossary/SystemsThinking.htm

  a discipline for seeing the 'structures' that underlie complex
  situations, and for discerning high from low leverage points.

https://thesystemsthinker.com/connecting-systems-thinking-and-action/

  The iceberg is one of the most frequently used models to explain
  systems thinking (see “The Iceberg”). Thanks to movies like “Titanic,”
  many people recognize that most of an iceberg sits hidden beneath the
  water; that is, out of sight.

https://www.accenture.com/us-en/blogs/blogs-devops-evan-erwee-system-thinking

  System Thinking allows people to make their understanding of social
  systems explicit and improve them in the same way that people can use
  engineering principles to improve their understanding of mechanical
  systems.

  ...

  We can therefore state that System Thinking for DevOps is required to
  understand Software Engineering as a social construct using
  engineering principles.

  From a System Thinking approach, DevOps thinking is fundamentally
  different from the traditional form of analysis. Traditional analysis
  focus on separating the individual software engineering pieces, e.g.,
  architecture, coding, testing, tooling etc. In contrast, the DevOps
  approach focuses on how all aspects of the software life cycle
  interact with other constituents of the system. Circling back to the
  definition of DevOps we can easily identify the ‘behavior traits’
  observed in the DevOps movement, namely: adaptable/flexible (improve
  relationship), cooperative (better communication), diplomatic
  (collaboration), etc.

http://www.systemicleadershipinstitute.org/systemic-leadership/theories/basic-principles-of-systems-thinking-as-applied-to-management-and-leadership-2/

  Systems thinking is a management discipline that concerns an
  understanding of a system by examining the linkages and interactions
  between the components that comprise the entirety of that defined
  system.

https://thesystemsthinker.com/introduction-to-systems-thinking/

  What exactly is systems thinking? In simplest terms, systems thinking
  is a way of seeing and talking about reality that helps us better
  understand and work with systems to influence the quality of our
  lives. In this sense, systems thinking can be seen as a perspective.
  It also involves a unique vocabulary for describing systemic behavior,
  and so can be thought of as a language as well. And, because it offers
  a range of techniques and devices for visually capturing and
  communicating about systems, it is a set of tools.

https://www.burgehugheswalsh.co.uk/Systems-Thinking/Approach.aspx

  Fundamentally, Systems Thinking is straightforward. We treat the
  situation of interest as a system and examine it using system
  properties and characteristics like purpose, context and structure to
  discover more.

http://study.com/academy/lesson/systems-thinking-in-management-definition-theory-model.html

  System thinking is a method of critical thinking by which you analyze
  the relationships between the system's parts in order to understand a
  situation for better decision-making. In simpler terms, you look at a
  lot of the trees, other plants and critters living around the trees,
  the weather, and how all these parts fit together in order to figure
  out the forest.


How does Docker container orchestration work?

welcome to DevOps SE, @bgarcial!

Well I think there are the following options depending on the use case:

Docker Container links are a according to the vendor site a legacy feature and you shouldn't use it
Docker compose should work well but you can't scale horizontally: the stack is runnable only one a single machine
With Docker Swarm using mostly same YML you can distribute your containers across many machines
If you are going to deploy large scale production environments you might want to choose Kubernetes over Swarm, maybe have a look for example on Rancher/Kubernetes.


Blue/green deployment with micro services and DNS

You should have 3 names when you're willing to do a blue/green deployment.

Two set of names, one for blue, one for green, which will works as is, and a production entrypoint which will point to either the blue name or green name.

The point of a blue/green deployment is to be able to test the full deploy before switching the clients entry point, so your services calls themselves don't have to use the entry point name itself.


Traefik configuration issue with local proxy

Make sure you connect your traefik instance to the client docker-network (as specified on the bottom of this page):

When running inside a container, Træfik will need network access through:

docker network connect <network> <traefik-container>


It is not possible for a ConfigMap to reuse or import other ConfigMaps.

You can, however, have many deployments reuse the same ConfigMap. In this case, you should extract out the common configuration to a single ConfigMap and reuse it across deployments while still allowing each individual deployment to have their own specific config.


How to build all projects in a monorepo by starting a build in Jenkins?

Build tools like maven or gradle could benefit you greatly here. It would do exactly what you have in mind as long as you have child project definition (with a build.gradle in gradle or pom.xml in maven).

I would not recommend the script as it would mean you have to maintain and tweak it in the future. Instead, investing in build tools can give you great ROI as they do much more than just build.

At the end if you cannot integrate a build tool, go for the script. It is a good idea also.


SaltStack states pattern and replace for multiple files under directory and for multiple line matches

You should use Jinja to loop over a list of files and pattern/replacement pairs.

{% set files = ['/a/foo.sh', '/b/bar.sh', '/b/baz.sh'] %}

{% for file in files %}
{% for p, r in [('/aaa/bbb', '/aaa/ccc'), ('/xxx/yyy', '/xxx/zzz')] %}
replace {{ p }} in {{ file }}:
   file.replace:
      - name: {{ file }}
      - pattern: {{ p }}
      - repl: {{ r }}
{% endfor %}
{% endfor %}

Keep in mind the identifiers need to be unique, that's why variables in replace {{ p }} in {{ file }}: are used.


Conversion tool for converting kubernetes yaml files from 1.15 to 1.17?

You can try the kubeyaml validator. It doesn't convert the yamls but it does show the exact errors for incompatible yaml for each version.


Is it possible to send email notification for Input (GO/NOGO) from a pipeline

You don't need any plugins.  You can send an e-mail from your Pipeline using the mail() step.  You can respond to the input step by using the REST API as documented in this answer on Stack Overflow.  Put the two together and you should be able to craft an e-mail that contains buttons for responding to the input.


What exactly is the cloud term 'a Firehose'?

The fire hose is a reference to an old Weird Al Yankovick movie, UHF (1989). In this movie, George Newman (portrayed by Yankovick) starts a local TV channel in the UHF band and is responsible for programming on the channel. After befriending the Station's Janitor, Stanley Spadowski, (portrayed by Michael Richards) George gives the Janitor a slot to run a show of his own making. The show, entitled "Stanley Spadowski's clubhouse" becomes an overnight sensation due to Spadowski's zany antics.

During a gameshow segment on the show, (video) Spadowski announces that a little boy, Joel Miller has "won" a contest by finding a marble in a wading pool of oatmeal and "gets to drink from the firehose!" - and this seems to be the likely origin of the term.

An oft-repeated sentiment in IT is that trying to comprehend a vast amount of material in a short amount of time (such as is the case when starting a new job, taking a training course or seminar on a technological subject or learning about a new product or device) is like "attempting to drink from a firehose" - it is nearly impossible to consume the entirety of such a large volume of information that is directed at you with such a rapid flow - a direct reference to the above scene.

It appears that in an attempt at humor, Cloudfoundry has chosen to call their Websocket "The Firehose" because websockets are used to "stream all application event data" (which they apparently regard as voluminous) most likely in reference to this sentiment/saying and scene.


How to start the latest version of k8s using minikube?

https://rominirani.com/tutorial-getting-started-with-kubernetes-on-your-windows-laptop-with-minikube-3269b54a226

user@host ~ $ minikube start --kubernetes-version="v1.7.5"
Starting local Kubernetes v1.7.5 cluster...
Starting VM...


Ansible can't recursively delete a directory

You may find the following format to be helpful:

- name: Recursively change ownership of a directory
file:
  path: /etc/foo
  state: directory
  recurse: yes
  owner: foo
  group: foo


No, jobs (which rust-latest and rust-nightly are examples of) will run in parallel if they are in the same stage (of which you only have one, build).


Dockerized MySql slower in Wifi than ethernet

your mysql server running on system A and if you are trying to make connection vai wifi or sending request to make connection on mysql it is mya possible reason should be latency in network. if both system is different and you have exposed you mysql to internet then also it may happen.


How to get branch hash using git command

git show -s --format="%H" <HEAD or Branch Name>

-s supressess the diff output
--format="%H" specifies that only the hash should be shown.
See man git-show for more details.


Try adding,- steup_remote_docker before - aws-ecr/build-image:


How do Django deployments happen on Servers?

Probably, you want need to go through something like Ansible first. Nothing is done manually these days, not even launching of EC2 instances or creating a VPC is manual and if it is, it needs to be changed immediately. You may also go through Terraform. You also need to understand CI/CD flow. So when any changes are pushed to git, a CI mechanism will execute which can include build, compile, unit test execution, integration test execution, code coverage reports etc which are all automated. Deployment can either be automated if CI is successful or on click of a button. Ansible can take care of the configuration changes which needs to be deployed including the application code changes. In fact, if you are in AWS, why don't you go for docker and deploy app in Docker using benefits of Amazon ECS. A Terraform code can do zero downtime deployments for you.
This is not specific to Django. It is used for any tech stack.


use different kubectl configs in different terminal sessions

Set the KUBECONFIG environment variable with a value that includes the relevant Kubernetes configuration file, the environment variable only applies to your current session/window.


Is there any way to get New Relic or PagerDuty to not alert during nightly maintenance?

We ended up creating a new PagerDuty service with a schedule attached to it.  That schedule only had anybody assigned for half of the day.  This ended up duplicating some info from the service we already had, but it was the least painful way we could find to deal with it.


How do i notify GitHub pull request for each stage of a parallel Jenkins pipeline?

I don't think I can help you with this, but I think I can suggest you where to look for a relevant information.

You can report any status check to your PR using Status API (https://developer.github.com/v3/repos/statuses/)
You can configure which Status checks are required from PR (https://help.github.com/en/github/administering-a-repository/about-required-status-checks)
Since pipeline is actually groovy code, calling Status API should not be a challenge. (Example with CURL - https://www.openmakesoftware.com/restful-json-api-calls-jenkins-pipeline/, example using native groovy - https://www.deployhub.com/native-groovy-api/)


How to recreate GCP Preemptible VMs behind LB before they expire?

Strictly you can't do this with a single preemptable virtual machine as you only get a 30 second grace period. While the majority of calls to the Google Compute Engine API would create a new device within 30 seconds, there are cases when this would fail due to network latency or temporary outage.

What you can do is create an Instance Group of Preemptible Virtual Machines with a minimum of two instances, the probability of GCE killing both at precisely the same time is quite low, and once GCE destroys one a new one will be created to bring the IG instance count back up to two.


AWS launch template create-before-destory?

It seems you're misunderstanding what the create-before-destroy lifecycle parameter of the launch configuration does.
It's about replacing the launch configuration within an autoscaling group, not replacing the autoscaling group.

Launch templates have versions which makes this parameter useless as you'll create a new version of the launch template before updating the ASG to use it.

Terraform gives an example in the ASG documentation.

The underlying idea is to be able to roll back more easily than with launch-configurations.


Which password database system work with ansible?

Any system would do. You might need to write a wrapper. For example:

> cat get-passwd-01.yml
- hosts: localhost
  gather_facts: no
  tasks:
    - command: ~/bin/get-passwd user9
      register: password
    - debug:
        msg: "passwd:{{ password.stdout }}"

.

> ansible-playbook get-passwd-01.yml | grep msg
    "msg": "passwd: T4gpJtZ69R5cZD9zQh"


Howto run test on test computer instead of build computer, using gitlab

Answering my own question here. "Multi project pipeline" is not necessary here. You simply have a single project and mark jobs with different "tags". You can then register runners for these different tags, on different machines.
(Artifacts are transferred from one job to the next the same, regardless if the runners are run on the same machine or on different ones)
https://docs.gitlab.com/ee/ci/yaml/README.html#tags


How to test database applications in QA testing environments?

The biggest problem with database backups is that you cannot simply snapshot them. You first have to make sure the DB is not mid-transaction when taking a snapshot, otherwise you will restore to an inconsistent state. In the case of windows, you can leverage the Microsoft SQL VSS writer to snapshot your database. You will want to be sure that when you take a snapshot of the VM, HyperV is capable of leveraging this VSS writer.

If you have a particularly large dataset, it may be better to leverage your storage filer to use delta-clones if your filer has this feature and it can help to address the size/time issue you are experiencing with backups.

In terms of deploying the new application to QA there are several ways to do this. One is to have an automated build system such as Jenkins, Travis CI or a similar tool automatically trigger a backup and deployment upon a code commit. This can be done by leveraging a configuration management tool such as SaltStack, Puppet or Chef (which all run on Windows) that can automatically set up and configure a SQL server for you and deploy your application if you are able to script a SQL install with PowerShell. Personally, I would recommend SaltStack as it's Reactor will allow you to broadcast a beacon when your backup/snapshot completes and trigger a build (Puppet lacks this capability).

One option for spooling up a VM each time is to leverage a system like The Foreman which has a HyperV plugin. You could then use SaltStack (vis-à-vis Reactor. This is probably the best way if you backup system blocks until the entire backup is complete, which I doubt) or your automated build system (Jenkins/Travis) interact with The Foreman's REST API to request a host be spun up. Once this comes up, it would automatically check in with your configuration management system to install SQL server and restore the snapshot in the new VM. If you combined this with an automated testing battery, you might even be able to implement a "Push on Green" system to repeat this process into your next environment if all the QA tests pass.

The advantage to using a piece of middleware like The Foreman is that it will allow you to deploy into any virtualization system or even the cloud and prevent you from being locked into HyperV and make your environment more portable.

I am sure there are other ways to get there, but this is one way I have used in the past to implement similar deployment workflows.


How to trigger Jenkins job by deploying git pipline using webhook?

I just played with it locally on a "safe network", but it may give you an idea.

I put in .git/hooks/post-commit something like:

wget --auth-no-challenge --http-user=$USER --http-password=$APITOKEN --secure-protocol=TLSv1 $JENKINS/job/$JOB/build?token=$JOBTOKEN -O /dev/null -o /dev/null

You need to create API token in jenkins and grab the webhook url for a given job.


How to include a custom Slack message only on completion?

As far as I was able to tell it's not possible.

We solved this by using Pipeline putting the custom messages in the Jenkinsfile.

The flexibility gained by having the Jenkinsfile in the git repo really helped our developers have a better understanding of the build process, and allowed them to make necessary changes to the build commands without the communications headache.

It's a nice excuse to move over to Pipeline.

It also happens to be very easy to do:

slackSend (color: colorCode, message: customMessage)


Is Cypress ok for acceptance tests on APIs in a CI/CD pipeline?

Cypress is designed for front-end end-to-end testing.  It is designed for testing a users interaction with your application and verifying that it works correctly (e.g. logging in with a valid username and password load the landing page).  Tools like this are designed to allow running in a headless mode which will let you run the tests in CI tool without loading a GUI.

Since it appears that your goal is to test the API interaction you are probably better off using a tool designed for that purpose.  Most API test tools allow you to send a request to your API and verify certain things about the response (i.e., response code, response headers, response bodies).

While using an end-to-end tool (like Cypress) will ultimately test your API, it will take longer to setup, run, and often be more brittle.  To use your above example, a tool like Cypress will require a script/test steps that log into your system, selecting multiple addresses for shipping, and verifying that everything displays properly.  A tool like Postman would just test that when you send a request to your FedEx API with a specific body and headers, you get the expected response, body, and headers back.  Setting up something like Postman will require a new tool, but will ultimately save time

Many of the API tools also provide a command line test runner and integrate with various CI tools (non-exhaustive list):

Postman - Free, relatively easy to setup, and includes an automated testing tool (newman).
SmartBear - Commercial product that integrates with most CI tooling.
Rest-Assured - Open source option for REST API testing.


In a Dockerfile, is there a way to avoid copying files to make them accessible to the RUN command?

You can do this with BuildKit and the experimental frontend. As of 18.09, BuildKit can be enabled with either:

export DOCKER_BUILDKIT=1

for a single shell, or to change the default for the host, you can add to /etc/docker/daemon.json:

{
  "features": {"buildkit": true}
}

You'll need to reload the docker engine after changing the above.

With BuildKit enabled, your Dockerfile would look like:

# syntax=docker/dockerfile:experimental

RUN --mount=type=bind,source=db-dump.gz,target=/tmp/db-dump.gz \
    zcat /tmp/db-dump.gz | mysql

You then build with a docker build . command as before, but with BuildKit enabled.

To see more about BuildKit's experimental features, see https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md


Does the "cattle not pets" distinction apply as equally to machine instances as to containers?

While technologically, containers and virtual machines are very different, there is no apparent difference from the perspective of your software. It seems like the argument in your question is that data is special and will always be a unique snowflake, so your question basically boils down to what to do about it in terms of DevOps, CI and Automation.

This is the perpetual challenge of the DevOps model. Ultimately, what you are asking is, either A) your data will be in a database or B) a datastore , so how do you manage that?

The answer is that yes, your database servers can and should also be treated as cattle - and I recently detailed several techniques and technologies for doing so in house. If you don't treat your data storage (databases, storage filers) as cattle you will find you have a single points of failure within your data(base) infrastructure and a lack of scalability and redundancy. These techniques I discuss in my linked answer basically allow you to distribute and cluster your relational database in a manner similar to Cassandra.

Managing your data(bases) and making data(bases) redundant is probably the most difficult challenge facing DevOps. The easiest way to solve the problem: outsource the headache to a cloud provider.

So in short, yes the "cattle not pets" distinction apply as equally to machine instances as to containers.


What are alternatives for Docker in Docker (dind) usage for Jenkins slave running in container?

To avoid relying on dind (out of building new images from a dockerfile which should be ok) I'd look around Jenkins's kubernetes plugin.

It obviously needs to bring to life a kubernetes cluster but allow to have a one time jenkins agent to run the test in a clean isolated environment, and then tear it off. The maintenance costs of the kubernetes cluster should not be greater than maintaining your Jenkins slaves.

The bonus point is that you can version you slaves with specific environment and don't have to tweak around for multiples version of ruby/java/python and let the task choose the proper "environment" by specifying the agent image to use and have a ready to test environment without any tweak pre-test.

I'd head toward kurbenetes as container scheduler because it is relatively lightweight to configure and maintain.


Use K8s Deployment Hostname In YAML Config

My idea of where the hostname was coming from was wrong. According to the docs:

  Every Service defined in the cluster (including the DNS server itself) is assigned a DNS name.

and

  Currently when a pod is created, its hostname is the Pod’s metadata.name value.

So the hostname is actually the name of the service. The last sentence in my question says I did not change the service in anyway. That was an unintentional lie. I did change the service name, or else selenium-hub as the hostname would have worked.

Changing HUB_PORT_4444_TCP_ADDR to the .metadata.name of the service got me closer. The connection was timing out, though.

I believed some of the configuration was wrong in my service yaml. So instead of using my yaml file, I created the service via:
kubectl expose deployment <deployment name here>

This created a service and I checked the selenium-hub and the node was finally connected. So there was an issue in my service yaml. To get the correct configuration I exported the currently working service:
kubectl get svc <service name here> -o yaml --export

I copied the exported configuration to my service yaml file and it also worked. Here was the difference between the service yamls

old yaml (copied from GitHub example above)
---
apiVersion: v1
kind: Service
metadata:
  name: selenium-hub-service
  labels:
    app: selenium-grid
spec:
  ports:
  - port: 4444
    targetPort: 4444
    protocol: TCP ------------- [added in new config]
    name: port0 ------------- [deleted in new config]
  selector:
    app: selenium-hub
  type: NodePort ------------- [deleted in new config]
  sessionAffinity: None

I am not sure why adding .spec.ports[0].protocol and deleting .spec.type and .spec.ports[0].name fixed the issue. If anyone could clarify in the comments I would appreciate it.


How do I edit my AWS ASG scheduled action useing the aws CLI

I just solved it using put-scheduled-update-group-action
aws autoscaling put-scheduled-update-group-action --auto-scaling-group-name temp_4_jenkins --scheduled-action-name training --start-time 2020-11-23T09:10:00Z --min-size 0 --max-size 0 --desired-capacity 0 --region eu-west-1 --output json


Why does TFS only support 1-to-many Predecessor /Successor relationships?

Because this link has direction from one side to another. Only one work item can on the start of this direction. However, work item:

Can has many successor
Can has many predecessor.

This type of link can not create the cycle, like:

Wi1 successor to WI2
Wi2 successor to WI1


System test frameworks for scale testing on physical and virtual systems

Test Kitchen? Your requirements are huge and vague but it's probably your best bet.


How the use Docker exec to update code into a running container

It's a bad practice.
You need to use Continuous Integration rebuild docker image (see Dockerfile) every time when source code changed.


Load balancing kubernetes cluster on budget KVM provider?

It depends on how much time you can invest into configuration and your configuration.

I have built my own cluster on AWS using Nomad and Consul where Fabio was used as Application Load Balancing layer (AWS ALB was lacking for my purposes).

Alternatively if you use k8s and do not need additional ALB you can use DNS RR (multiple A or AAAA records for your domain) or create HAProxy cluster in front of your application.


Depends on where the source data is when considering how you want to initialise the database.

If the data pre-exists in another SQL-Azure Database you could create a BACPAC and export the file to a blob store in Azure. Then using PowerShell in your pipeline import the data into your created database. Bare in mind this will be time costly dependent on the amount of data and the size of the database.
If the data is only small, an approach we have used before to build and populate tables is a DACPAC. Here we had the create scripts for our tables and then stored procedures with static reference data stored in insert statements which were run after the DAC built the tables & indexes etc. Here is an example in how to create a DACPAC project in Visual Studio and populate from an exisiting database. Alternatively if you're building from scratch you can add the required database objects to the solution and create them using SQL Statements.
The third option would be streaming data from a source using an Azure Function and blob storage however without knowing more about your set up I can't advise more.

Hope that helps


Most likely the port is published on the VM and you are trying to connect to your laptop's IP and not seeing it. Newer versions of docker that use xhyve include some networking tricks to publish the port all the way to your laptop, but docker-machine with VirtualBox does not do this.

You can lookup the IP of your VM with docker-machine ls or see it in the variable echo $DOCKER_HOST and connect to port 8000 on that IP to reach your application.


we do this with .conf files and loading them via compose

env_file: /path/to/our.conf

they can also be used with

docker run --env-file /path/to/our.conf ourcontainer:version

In some cases we use the variables defined in our conf files directly, meaning the code we are running reads them direct, in others we need to modify other files on a per environment basis so we use sed or other tools to make changes to a file we have turned into a template


cannot get bash prompt: Error response from daemon: Container foo is not running

Whoever created basex/basexhttp:latest does not seem to know Docker very well. They copied files necessary to a directory that was later configured as a volume, which means it can only be bound to an anonymous volume, otherwise files inside will be removed. They left source code inside the image, they used a single stage build with maven:3-jdk-8-alpine as base image, which means the final image has many unnecesary things installed. They left the entrypoint from the base image intact, which I guess is why they needed to create an empty /srv/.m2 directory and set it as Maven config dir for the image to start. It switches user from root to basex, which causes permission problems with bind mounts. Generally it's a mess.
I see two ways to successfully run this image:
First:
docker run -d \
    --name basexhttp \
    --publish 1984:1984 \
    --publish 8984:8984 \
    --entrypoint "" \
    basex/basexhttp:latest /usr/local/bin/basexhttp

This uses only anonymous volumes and clears that unnecessary entrypoint.
You can stop it with docker stop basexhttp and then remove it with docker rm -v basexhttp. That -v is important - it removes the volumes - unless you memorize their very long IDs and which was mounted which you won't be able to reuse them anyway, so you may as well remove them from your system.
Second:
mkdir -p $HOME/basex/data
mkdir -p $HOME/basex/repo
mkdir -p $HOME/basex/webapp
chmod a+w basex
sudo chown -R 1984:1984 $HOME/basex
docker run -d \
    --name basexhttp \
    --publish 1984:1984 \
    --publish 8984:8984 \
    --volume "$HOME/basex/data":/srv/basex/data \
    --volume "$HOME/basex/repo":/srv/basex/repo \
    --volume "$HOME/basex/webapp":/srv/basex/webapp \
    --entrypoint "" \
    basex/basexhttp:latest bash -c "cp -r /usr/src/basex/basex-api/src/main/webapp/WEB-INF /srv/basex/webapp && exec /usr/local/bin/basexhttp"

This creates the necessary directories and sets their permissions so that both your user on the host and user.
You can stop it as in the previous example and remove with docker rm basexhttp. This time -v is not necessary, as there are no anonymous volumes.
You can verify that the contaienr is running correctly by executing:
docker exec -ti basexhttp basexclient

Username and password are both admin.


How to "docker-compose build" a service from code located in local git repository?

What you are specifically after (building from a bare repository on a local filesystem) isn't functionality offered by Docker, and by extension, docker-compose. Docker supports building from a few different URLs, but not bare repositories on local filesystems. You can try a few workarounds:

Build straight from Github: docker build https://github.com/docker/rootfs.git
Run a Git daemon to use the git:// protocol: git daemon --base-path=. --export-all (in /home/nou/code)
Running a Git daemon from within Docker, mounting your bare repository as a volume and building inside it via a mounted Docker socket
Converting your bare repository to a normal one

https://docs.docker.com/engine/reference/commandline/build/#build-with-url


What you're after is an Binary repository manager

Quoting from Wikipedia with added links:

  Notable Universal package managers include:[6]

  Apache Archiva
  Jfrog's Artifactory
  Inedo's ProGet
  Sonatype Nexus

I know for sure Nexus and Artifactory match your requirements (even if proper UI is a bit subjective and you mileage may vary).


Unable to send cloudwatch logs to loggly using a blueprint lambda - Invalid ciphertext

Follow these steps to fix InvalidCiphertextException:

Open the Lambda function in AWS console.

Scroll down to the Environment Variables section & click Edit:

Delete the value of kmsEncryptedCustomerToken & replace it with the customer token you got from https://<your-company>.loggly.com/tokens. The token looks like this: 72cf6d64-256e-449d-aabd-49e1f422ed58.

Expand the Encryption Configuration section at the bottom.

Select Enable helpers for encryption in transit.

Click the Encrypt button that appears next to kmsEncryptedCustomerToken.

Select the logglyCustomerToken KMS key (CMK) & Encrypt.

Finally, Save the environment variables.

The Lambda function test should succeed now. If you face any other issues or need a complete end-to-end screenshot-guided tutorial for the entire process of sending CloudWatch Logs to Loggly, please see my blog post.


How to deploy a custom build of a Jenkins core plugin?

This section of the official Jenkins Plugin Tutorial will probably be of use.

According to the writers of that documentation, you need to copy your .hpi file to $JENKINS_HOME/plugins, remove the plugin's development directory, and finally create an empty file called .hpi.pinned for maven to use. Then you should be ready to restart Jenkins and use your custom build of a core plugin.

They also suggest running mvn clean when you're ready to update Jenkins to remove old files (after changing core plugins, old .jar files might still be around after updating).


You can use Terraform for azure to handle the dynamic creation of azure resources in the release pipeline in Azure DevOps. Terraform maintains a state file for all the created resources after a run.

If the script is run multiple times, it ignores already created resources unless there are any properties updated and created new resources.

FYI, follow below link for a quick example
Terraform example


How to get dependency from file share in Azure DevOps pipeline

Steps for download are described here: How To Download Azure BLOB Storage Using Azure PowerShell. According to this article, you have to use the connection string from the Access keys page. You can create your custom secret variable (Variables) and save a connection string in your new variable. Then in your ps scripts you can reference to connection string (stored in variable) with "$env:VARIABLE_NAME".


How do I dynamically take variables from previous plays in Ansible?

It feels dirty, but I guess something like that would work:

- name: ec21
  [...]
  register: ec21_result

- set_fact: end_result= "{{ ec21_result }}"
  when: ec21_result|succeeded

- name: ec22
  [...]
  register: ec22_result
  when: ec21_result|failed

- set_fact: end_result= "{{ ec22_result }}"
  when: ec22_result|succeeded

- name: ec23
  [...]
  register: ec23_result
  when: ec22_result|failed

- set_fact: end_result= "{{ ec23_result }}"
  when: ec23_result|succeeded

- name: Add new instance to host group
  add_host:
    hostname: "{{ item.private_ip }}"
    groupname: launched
  with_items: "{{ end_result.instances }}"

Basically setting the end_result as soon as you succeed a play.


How to share RDS snapshots across account roles?

To my knowledge there are only Tag based Iam policies available to limit access to resources for certain users or user groups.

Because of this, you should simply be able to tag an RDS snapshot both production and development for you to access it with your development user or group.

There are a large number of security considerations to think through, so don't undermine the security setup. I would therefore expect that only manual tagging is appropriate for this specific use case


What I have done before goes on the same direction as Zeitounator commented; you can have one or more "meta" or master playbooks that import the ones you need to run your entire automation. To add to Zaito's comment; you can use the when clause on import_playbook or include to control the execution of those "inner" playbooks; something like this:
Say you want to execute PlatformB playbooks only for an stg or prod environment using a defined env ansible variable:
- include: platformA/do_something_on_platformA.yml
- include: platformB/do_something_on_platformB.yml
  when: env == 'stg' or env == 'prod'

you could change the value of env using --extra-vars on execution time or something.
hope it helps, cheers.


What's the term for moving/promoting code between deployment environments

I don't think there is a generic term for promoting to production. I can tell you that within the devops community, terms are frequently interchanged. As long as the point being addressed is well thought out and made clear, the point can be understood. Some of the terms that are used when referring to the movement between environments are deploy, propagate, move, promote, and release. I will say though, whenever I hear the term stage, I usually associate it with a pre-production environment. I think most others would assume the same.

Within your organization/groups, I think it is important to try to be consistent with your terminology. This eliminates any confusion and assumptions people can make. An example would be, in my organization, we use the term deploy whenever we are referencing moving between environments except when referring to the production environment which we use the term release. This is to emphasize that it is the final movement in the pipeline/process.


Join a Linux Docker swarm with a local Windows Docker for test purposes

Why do you want to create a new Manager node, risking damaging your raft consensus and cie?
I find it way easier to simply expose the docker socket locally, as if you were working inside the node, but with your Windows environment.
To do so, simply open a ssh tunnel that exposes /var/run/docker.sock:
 ssh -M -S ~/.docker.sock \
     -fnNT -4 -L localhost:1337:/var/run/docker.sock \
     USER@MANAGER_IP

Refer to man ssh to see what all these options mean.
It will open an ssh tunnel; you will still be in your local environment shell, only thing left to do is set the proper DOCKER_HOST environment var, so your docker cli is bound to your swarm manager.
export DOCKER_HOST=localhost:1337
And that's it.
Note: Docker also includes an option to do this; I've quickly searched for it but can't manage to find it. It's in the most recent version of docker.


How to write logs for certain tasks only?

It's in the wish list, I think. Quoting from the doc log_plays - write playbook output to log file

* This callback writes playbook output to a file per host in the /var/log/ansible/hosts directory
* TODO: make this configurable

IMHO required functionality is not available at the moment. Take a look at the available plugins. "You can only have one plugin".

ansible-doc -t callback -l

There is an option to customize a callback. context_demo looks promising.


Your question is quite vague/generic.

People store their source code (or generally, files) in git; let Jenkins run jobs (like CI/CD pipelines), use Docker to put their software into well-defined containers, Nagios to monitor their systems, ELK to collect, store and visualize (mostly logging, event) data, AWS to host their stuff, and Chef to manage their configurations.

If you are studying that and learning about it, then you should have gotten that kind of information, really. The scenario in real world companies is simply to use those tools for what they are made, in different combinations as needed.

If you really have just gotten some deficient information from your college or wherever you studied those things, then get a book like The DevOps Handbook and give it a read. It should clear up how all of this would work together in the real world.